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Abstract 

Publicized  criticism  of  Air  Force  cost  estimates  assert  the  Air  Force  produces 
program  cost  estimates  that  drift  towards  mediocre  guesses  compared  to  the  high 
fidelity  instruments  of  time  and  cost  intended.  While  many  researchers  have  sought 
to  identify  the  sources  for  cost  and  schedule  growth,  most  researchers  have  failed  to 
analyze  the  resource  utilization  of  the  cost  community.  This  research  explores  how 
the  cost  community  allocates  its  time.  Furthermore,  by  examining  how  resources 
are  spent,  this  research  juxtaposes  the  desires  of  recent  Congressional  and 
Department  of  Defense  policies  against  the  current  demands  of  the  cost  community. 
A  thorough  understanding  of  resource  allocation  requires  research  into  the  inherent 
virtualness  of  the  community.  Early  virtualness  predicated  the  notion  of  extremes, 
either  virtual  or  not.  However,  recent  literature  expands  virtualness  into  gradients 
and  explains  that  all  teams  display  some  measure  of  virtualness.  Unfortunately, 
scholars  currently  debate  the  basic  definition  of  virtualness  as  being  comprised  of 
either  three  or  four  individual  dimensions.  This  research  uses  an  Internet-based 
questionnaire  to  ascertain  a  measure  of  virtualness.  The  findings  of  this  research 
support  a  four-dimension  measure  of  virtualness.  This  research  uses  structural 
equation  modeling  to  validate  and  test  for  good  reliability  of  the  created  13 -item 
measure  for  virtualness.  This  research  finds  that  the  creation  and  modification  of 
cost  estimates  consumes  the  majority  of  resources,  while  the  cost-estimating 
community  spends  few  resources  on  the  implementation  or  follow-up  of  estimates. 
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VIRTUALNESS  OF  THE  COST  ESTIMATING  COMMUNITY 


I.  Introduction 


Overview 

Department  of  Defense  [DoD)  acquisition  programs  tend  to  cost  more  and 
require  longer  development  periods  than  initially  estimated  [Bolten,  Leonard, 

Arena,  Younossi,  &  Sollinger,  2008].  Some  argue  that  Air  Force  cost  estimates  drift 
towards  mediocre  guesses  compared  to  the  intended  high  fidelity  instruments  of 
time  and  cost.  Many  researchers  have  powered  much  deliberation  and  investigation 
seeking  to  identify  the  sources  for  cost  and  schedule  growth.  Differing  opinions 
have  surfaced  claiming  insufficient  resources,  requirements  creep,  or  inadequate 
training  as  possible  drivers  of  cost  growth  (Bolten  et  al.,  2008].  While  many 
researchers  have  sought  to  identify  the  sources  for  cost  and  schedule  growth,  most 
have  failed  to  analyze  the  resource  utilization  of  the  cost  community.  This  research 
explores  how  the  cost  community  allocates  its  time  and  resources.  Furthermore,  by 
examining  how  resources  are  spent,  this  research  will  juxtapose  the  desires  of 
recent  policy  changes  and  the  current  demands  within  the  cost  community. 

In  addition  to  resource  allocation,  a  complete  understanding  of  the  cost 
community  relies  on  an  understanding  of  its  inherent  virtualness  environment. 
Virtualness  refers  to  the  composition  of  distance,  reliance  on  technology,  value 
provided  by  technology  and  synchronicity  of  interactions.  These  four  primary 
dimensions  contribute  to  the  level  of  virtualness  (Kirkman  &  Mathieu,  2005;  Griffith, 
Sawyer,  &  Neale,  2003].  Among  many  moderating  aspects  of  team  effectiveness. 
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virtualness  correlates  with  many  potentially  detrimental  factors  [Griffith  et  al., 
2003], 

Recent  disasters  on  the  Gulf  Coast  highlight  the  potential  effects  of 
virtualness.  Communication  breakdowns,  information  technology  failures,  and 
misinterpretations  are  three  major  contributing  factors  identified  as  interfacing 
with  the  Hurricane  Katrina  relief  efforts  [Garnett  &  Kouzmin,  2007].  More  recently, 
the  British  Petroleum  [BP]  Deepwater  Horizon  explosion  contained  aspects  of 
virtualness  in  multiple  ways.  The  explosion  and  resulting  oil  leak  involved  teams  of 
physically  separated  people  heavily  reliant  on  technology  and  operating  the  well  at 
extreme  depths  in  the  ocean.  A  situation  exhibiting  the  four  dimensions  of 
virtualness.  The  equipment  needed  for  capping  the  well  relied  solely  on  technology 
as  the  depth  prevented  a  human  from  physically  touching  the  well.  The 
maintenance  needed  on  the  well  required  a  remote  operator  performing  intricate 
procedures  miles  away.  The  controller  relied  on  the  information  provided  by 
technology,  as  well  as  the  value  of  the  information  returned  through  the  remote 
cameras.  The  equipment  and  actual  capping  process  demanded  synchronicity,  for 
any  uncontrolled  delay  between  operator  input  and  equipment  action  could  create 
unintended  consequences.  The  BP  disaster  presents  an  unwelcome  opportunity  to 
study  the  potentially  detrimental  consequences  of  virtualness.  However,  this  paper 
is  not  a  case  study  of  virtualness  as  it  relates  to  cataclysmic  disasters. 

Traditional  research  into  virtualness  viewed  teams  as  traditional  or 
completely  virtual  [Anderson,  McEwan,  Bal,  &  Carletta,  2007].  As  the  academic 
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community  accepted  virtualness,  researchers  started  applying  levels  of  virtualness 
to  team  dynamics.  Hypotheses  stated  that  companies  were  seldom  wholly 
traditional  (non-virtual]  or  completely  virtual.  Most  organizations  maintained  some 
level  of  virtualness.  These  hypotheses  limited  virtualness  as  a  level  of  an 
organization  or  product.  However,  recent  research  into  virtualness  progresses 
towards  gradients  of  virtualness  not  being  limited  to  organizations  or  products. 
Virtualness  not  only  varies  within  an  organization,  but  also  may  vary  at  a  team  or 
even  personal  level. 

Virtualness  as  a  management  concept  is  still  in  its  infancy,  with  the  bulk  of 
published  research  occurring  within  the  last  20  years.  While  the  knowledge  base  of 
virtualness  continues  to  expand,  the  debate  as  to  the  exact  elements  comprising 
virtualness  have  solidified  around  three  dimensions,  with  a  fourth  dimension  being 
argued  by  many  (Kirkman  &  Mathieu,  2005].  While  these  dimensions  have  gained 
acceptance  as  accurate  indicators  for  the  level  of  virtualness,  the  researchers  has  yet 
to  develop  a  valid  and  reliable  measurement  for  virtualness.  This  research  aims  to 
build  an  accurate  measurement  for  understanding  the  degree  of  virtualness 
exhibited  at  an  individual  level.  The  measurement,  while  developed  in  an  Air  Force 
community,  should  apply  generically  to  all  organizations. 

Purpose 

Developing  a  virtualness  measure  has  greatly  enhanced  my  research  into  the 
organization  of  the  cost  community.  This  research  effort  intends  to  establish  a 
snapshot  for  the  Air  Force  cost  estimating  community.  Understanding  the 
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virtualness  inherent  within  organizations  allows  decision  makers  to  adjust 
leadership  style  and  policy  to  enhance  effectiveness.  The  complexity  and 
hierarchical  nature  of  the  Air  Force  forces  a  certain  amount  of  virtualness,  yet 
virtualness  establishes  threats  to  effectiveness.  Research,  however,  indicates  that 
strong  transformational  leadership  can  overcome  many  of  the  detrimental  aspects 
of  virtualness  (Hertel,  Geister,  &  Konradt,  2005).  Understanding  the  potential 
effects  of  virtualness  married  with  the  given  level  of  virtualness  apparent  within  a 
team,  arms  leadership  with  the  opportunity  to  guide  resources  towards  critical 
needs. 

In  addition  to  establishing  a  measure  for  virtualness,  this  research  aims  at 
identifying  asset  utilization  in  the  cost  estimating  community  with  respect  to 
personnel.  Recent  policies  aimed  at  controlling  program  costs  stress  the 
importance  of  accurate  cost  estimates.  As  a  result,  the  Air  Force  cost  community  is 
transforming  and  adapting  to  the  increased  pressures  and  demands.  While 
leadership  has  stressed  the  need  for  the  revitalization  of  the  acquisition  community, 
little  understanding  exists  about  its  implementation  or  acceptance  at  the 
organizational  level.  This  research  utilizes  an  Internet-based  questionnaire 
presented  to  cost  estimators  in  hopes  of  obtaining  a  more  thorough  appreciation  of 
the  allocation  of  time.  In  addition  to  time  allocation,  the  questionnaire  probes  into 
experience  levels,  training,  and  professional  certificates  of  the  military  and  civilian 
cost  estimators.  This  research  presented  a  similar  but  more  open-ended 
questionnaire  to  supervisors.  The  supervisory  questionnaire  identifies  how  the 
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team  as  a  whole  allocates  its  time.  The  questionnaire  also  explores  recent  personnel 
and  office  changes  subsequent  of  policy  implementation. 

The  questionnaire  provides  a  current  snapshot  of  the  cost  estimating 
community.  This  research  compares  the  findings  against  a  similar  census  conducted 
by  the  RAND  Corporation  in  2008  (Vernez  &  Massey,  2009].  Chapters  4  and  5 
discuss  and  highlight  potentially  enlightening  comparisons  between  the  studies.  In 
addition  to  the  comparison,  1  also  discuss  the  questionnaire  findings  for  resource 
allocation,  training,  and  virtualness  in  Chapters  4  and  5. 

Study  Context 

The  effort  to  establish  a  reliable  measure  of  virtualness  centers  on  an 
individual  level  survey  of  approximately  400  Air  Force  cost  personnel.  The  Air 
Force  primarily  centralizes  the  cost  community  within  three  organizational  areas. 
The  Air  Force  Cost  Analysis  Agency,  headquartered  in  Washington,  D.C.,  occupies 
the  focal  point  of  all  cost  analysis  policy  and  acts  as  an  independent  cost  review  for 
major  defense  acquisition  programs  (MDAPs].  The  remaining  centers  of  cost 
analysis  lie  at  the  two  major  commands  [MAJCOMs],  Air  Force  Material  Command 
(AFMC]  and  Air  Force  Space  Command  (AFSPC].  These  MAJCOMs  function  as  the 
acquisition  centers  for  MDAPs  for  the  Air  Force.  The  hierarchal  structure  of  the 
military,  combined  with  the  geographic  separation  between  the  Air  Force  Cost 
Analysis  Agency  and  MDAP  acquisition  hubs,  creates  an  excellent  opportunity  to 
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gather  the  information  necessary  to  both  establish  a  measure  and  build  a  baseline 
understanding  of  virtualness. 

Research  Questions 

This  thesis  addresses  the  following  research  questions: 

Primary  Research  Question:  Is  there  a  disparity  between  leadership’s 
expectations  and  employee’s  activities,  which  affect  the  implementation  of 
the  Weapons  Systems  Acquisition  Reform  Act  of  2009  and  the  Acquisition 
Improvement  Plan? 

Secondary  Research  Question  1:  What  is  the  current  allocation  of  time 
within  the  cost  estimating  community  supporting  acquisition  reforms? 

Secondary  Research  Question  2:  Does  virtualness  affect  the  Air  Force  cost 
estimating  community’s  ability  to  sufficiently  support  acquisition  reform? 

Hypotheses 

This  thesis  addresses  the  following  hypotheses  in  support  of  the  previously 
mentioned  research  questions: 

Hypothesis  1:  Virtualness  is  negatively  correlated  to  trust. 

Hypothesis  2:  Virtualness  is  negatively  correlated  with  job  satisfaction. 
Hypothesis  3:  Virtualness  is  negatively  correlated  with  organizational 
commitment. 

Hypothesis  4:  Virtualness  is  negatively  correlated  with  turnover  intention. 
Hypothesis  5:  Trust  is  negatively  correlated  with  turnover  intention. 
Hypothesis  6:  Job  satisfaction  is  negatively  correlated  with  turnover 
intention. 
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Hypothesis  7:  Organizational  commitment  is  negatively  correlated  with 
turnover  intention. 

Hypothesis  8:  Trust  mediates  the  correlation  between  virtualness  and 
turnover  intention. 

Hypothesis  9:  Job  satisfaction  mediates  the  correlation  between  virtualness 
and  turnover  intention. 

Hypothesis  10:  Organizational  commitment  mediates  the  correlation 
between  virtualness  and  turnover  intention. 

Organization  of  Thesis 

The  primary  purpose  of  this  paper  is  to  establish  a  baseline  measurement  of 
virtualness  of  the  cost  community  within  the  United  States  Air  Force.  However,  to 
develop  understanding,  1  first  discuss  some  of  the  issues  currently  facing  the  cost 
community.  This  discussion  includes  recent  changes  undertaken  by  leadership  to 
curb  the  dramatic  growth  in  cost  as  shown  by  MDAP  reports.  In  Chapter  2, 1  discuss 
some  of  the  published  literature  relating  to  the  contents  of  this  thesis.  Chapter  3 
includes  the  methodology  used  to  gather  and  analyze  the  data.  In  Chapter  4, 1  report 
the  findings  of  my  research.  Lastly,  in  Chapter  5, 1  discuss  the  findings  and  the  ways 
leadership  can  utilize  the  results  to  benefit  the  cost  community. 
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II.  Literature  Review 


Chapter  Overview 

Chapter  II  highlights  some  acquisition  reform  initiatives  primarily  focusing 
on  the  Weapon  System  Acquisition  Reform  Act  [WSARA]  of  2009  and  the  Air  Force 
Acquisition  Improvement  Plan.  This  chapter  also  reviews  research  into  the 
effectiveness  of  acquisition  reforms  and  studies  into  the  Air  Force  acquisition  cost 
workforce.  While  there  is  a  great  deal  of  research  into  the  effectiveness  of  reform 
initiatives,  dissention  among  the  conclusions,  as  well  as  problems  with  data  and 
definitions,  weakens  the  impact  of  the  research.  This  chapter  discusses  the  current 
research  into  the  acquisition  community— the  RAND  study,  among  others— and 
discusses  some  of  its  weaknesses  and  limitations.  Next,  I  present  literature  on 
virtualness  and  the  way  my  research  establishes  a  more  complete  knowledge  of  the 
community  and  add  insight  into  more  effectively  implementing  reform.  Lastly,  this 
chapter  highlights  current  trends  in  management  studies,  which  promote  a  greater 
understanding  into  the  utilization  of  cost  personnel.  Understanding  the  information 
within  this  chapter  promotes  a  greater  understanding  and  logical  flow  for 
subsequent  information  and  the  conclusions  presented  within  later  chapters. 

Acquisition  Reform 

The  Department  of  Defense  suffers  a  long  history  of  acquisition  problems  and 
errors.  While  some  issues  result  from  single  individuals  manipulating  opportunities 
in  selfish  ways,  as  with  the  Darleen  Druyun  case,  many  problems  surround  major 
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acquisition  programs  as  a  whole.  Two  detrimental  problems  of  acquisition,  cost 
overruns  and  schedule  delays.  Both  problems  often  result  in  the  delivery  of  fewer 
weapon  systems  to  the  warfighter  behind  schedule  and  at  an  increased  cost  per 
item.  Negative  results  in  major  system  acquisitions  draw  Congressional  interest.  In 
order  to  counteract  detrimental  trends  in  MDAPs,  acquisition  reforms  pass  through 
Congress  at  an  alarming  rate,  culminating  most  recently  in  the  Weapon  Systems 
Acquisition  Reform  Act  [WSARA]  of  2009:"The  purpose  of  this  law  will  be  to  limit 
cost  overruns  before  they  spiral  out  of  control”  [President  Barak  Obama  at  signing 
of  WSARA  Legislation].  Signed  into  law  on  May  22,  2009,  WSARA  aims  at  lofty 
improvements  in  major  defense  acquisition  programs  for  the  entire  DoD.  The 
overarching  policy  creates  new  government  positions  among  other  aspects,  seeking 
to  reduce  cost  overruns.  One  of  the  many  major  changes  requires  MDAPs  to 
undergo  a  thorough  preliminary  design  review  before  Milestone  B.  The  total 
ramifications  from  this  single  policy  change  are  estimated  at  being  numerous  and 
drastic;  however,  much  of  the  ripple  effect  is  unknown. 

A  recent  policy  targeting  the  Air  Force  specifically  is  the  Acquisition 
Improvement  Plan  [AlP].  Signed  May  4,  2009,  by  Chief  of  Staff  General  Norton  A. 
Schwartz,  the  AlP  aims  at  "recapturing  acquisition  excellence  by  rebuilding  an  Air 
Force  acquisition  culture  [Office  of  the  Assistant  Secretary  of  the  Air  Force 
[Acquisition],  2009].  The  plan  states  that  many  challenges  face  AF  acquisition  and 
identifies  specific  actions  to  counteract  negative  trends.  While  the  plan  summarizes 
five  critical  areas  for  improvement,  one  specifically  relates  to  the  underlying 
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purpose  of  this  thesis:  "unclear  and  cumbersome  internal  Air  Force  organization  for 
acquisition  and  Program  Executive  Officer  [PEO]  oversight"  [Office  of  the  Assistant 
Secretary  of  the  Air  Force  [Acquisition],  2009].  The  previous  statements  drive  at 
the  core  of  this  effort:  to  understand  the  current  cost  acquisition  workforce  in  order 
to  help  decision  makers  effectively  utilize  limited  resources. 

The  sheer  number  of  acquisition  reform  initiatives  highlights  the  dire 
condition  of  MDAPs.  Table  1  includes  some  of  the  major  reform  efforts  to  include 
policy  changes  and  implement  special  commissions  striving  at  improving 
government  acquisitions. 

Table  1:  Acquisition  Reform  Initiatives 


Acts 

Commissions 

*Arnned  Services  Procurement  Act  of  1947 

*1949  Hoover  Commission 

*  Federal  Procurement  Policy  Act  of  1948 

*1955  Hoover  Commission 

*Federal  Procurement  Policy  Act  Amendment  in 

*1969  Fitzhugh  Commission 

1978  established  the  Federal  Acquisition 

*1972  Commission  on  Government  Procurement 

*Goldwater-Nichols  Department  of  Defense 

*1981  Carlucci  Initiatives 

Reorganization  Act  of  1986 

*1982  Grace  Commission 

*Government  Performance  and  Results  Act  1993 

*1986  Packard  Commission 

*Federal  Acquisition  Streamlining  Act  of  1994 

*1989  Defense  Management  Report 

Why  Reform 

Central  to  the  effort  behind  the  numerous  reform  initiatives  lies  acquisition 
systems  cost  growth  and  schedule  delay.  Major  defense  acquisitions  systems  costs 
grow  at  the  alarming  rate  of  over  45%  at  milestone  B  on  average  [Arena,  Leonard, 
Murray,  &  Younossi,  2006].  Superficially,  the  previous  statement  rings  of  logic  and 
understanding;  however,  what  is  cost  growth  or  schedule  delay?  Is  the  idea  that  a 
weapon  system  costs  more  than  initially  expected  cost  growth?  How  often  does  the 
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DoD  purchase  a  weapon  system  initially  designed  instead  of  an  improved  iteration 
that  costs  more  yet  has  greater  capabilities?  The  statement  regarding  cost  growth  is 
synonymous  with  schedule  delay.  One  academic  definition  of  cost  growth  is  "the 
ratio  between  the  most  recent  selected  acquisition  report  (SAR]  estimate  and  the 
cost  estimate  baseline  reported  in  a  prior  SAR  issued  at  the  time  of  a  given 
milestone"  (Younossi,  Arena,  Leonard,  Roll,  Ir.,  Jain,  &  Sollinger,  2007J.  In  the  past 
30  years,  cost  growth  associated  specifically  with  the  development  phase  of  MDAPs 
largely  remained  constant  (Younossi  et  al,  2007].  A  RAND  study  published  in  2008 
identifies  program  decisions  as  the  primary  source  of  cost  growth  [Bolten  et  al. 
2008J.  Changing  requirements,  quantity,  or  other  decision  factors  account  for  over 
two-thirds  of  all  cost  growth  [Bolten  et  al.,  2008].  Often,  blame  for  cost  growth  gets 
pushed  towards  the  realm  of  cost  estimators;  however,  as  indicated,  two-thirds  of 
cost  growth  is  outside  the  estimators’  control.  Not  completely  devoid  of  blame,  the 
cost  estimating  community  accounts  for  approximately  one-fourth  of  total  cost 
growth  (Bolten  et  al.,  2008J.  Acquisition  reform  addresses  the  cost  estimating 
personnel  due  to  inaccurate  cost  estimates  accounting  for  10.1%  increase  in  MDAP 
cost  (Bolten  et  al.,  2008).  However,  the  majority  of  reports  indicating  cost  growth 
derive  data  from  SARs.  Legally  mandated  and  heavily  utilized  by  decision  makers 
for  budgetary  decisions,  the  SAR  is  not  without  problems  [Hough,  1992).  Table  2 
highlights  the  most  notable  problems  that  surround  utilizing  the  SAR  for  data 
purposes. 


11 


Table  2:  Notable  Problems  of  SAR  (Hough,  1992) 


Failure  of  some  programs  to  use  a  consistent  baseline  cost  estimate 
Exclusion  of  some  significant  elements  of  cost 

Exclusion  of  certain  classes  of  major  programs _ 

Constantly  changing  preparation  guidelines _ 

Inconsistent  interpretation  of  preparation  guidelines  across  programs 
Unknown  and  variable  funding  levels  for  program  risk _ 

Cost  sharing  in  joint  programs _ 

Reporting  of  effects  of  cost  changes  rather  than  their  root  causes 


The  problems  originating  through  the  use  of  an  SAR  for  data  analysis  increase  the 
need  for  thorough  understanding  by  decision  makers.  Careful  analysis  and 
compensation  techniques  mitigate  some  risks  and  errors;  however,  any  conclusions 
drawn  from  SAR  reports  must  include  necessary  caveats  to  warn  readers  [Hough, 
1992].  In  summary,  between  the  fundamental  problems  of  defining  true  cost 
growth  and  the  inherent  errors  contained  in  utilizing  SARs  to  produce  growth 
estimates,  the  cost  community  must  cautiously  approach  all  reforms  with 
knowledge  and  understanding. 

Effectiveness  of  Acquisition  Reform 

The  sheer  number  of  different  reforms  screams  of  a  lack  of  effectiveness.  The 
goal  of  many  reform  initiatives  is  to  strengthen  the  acquisition  community,  control 
costs,  and  reduce  schedule  delays.  The  magnitude  and  scope  of  the  Department  of 
Defense  complicates  the  implementation  of  reforms  [Cooper  &  Rumbaugh,  2009]. 

As  such,  much  research  seeks  to  understand  how  effective  the  acquisition  reforms 
are  when  finally  implemented.  A  few  problems  arise  when  trying  to  measure  both 
implementation  and  effectiveness. 
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Implementing  Congressional,  DoD,  or  AF  level  policy  changes  require 
numerous  geographically  separated  personnel  to  adjust  standard  work  practices. 
The  number  of  personnel  alone  is  not  the  greatest  difficulty.  Policy  interpretation 
yields  disparity  between  organizations  and  individuals.  The  most  mundane  of 
changes  must  be  coordinated  amongst  multiple  stakeholders,  all  of  whom  provide 
guidance  as  to  the  interpretation  of  the  policy.  Senior  leadership  seeks  to  alleviate 
the  interpretation  disparity  through  guidance  memorandums,  which  also  require 
interpretation.  At  no  point  is  it  possible  to  completely  remove  differences  in 
interpretation  (Radin,  1999).  In  addition  to  actual  interpretation  issues,  omissions 
or  contradictions  amplify  the  difficulties  inherent  in  policy  guidance  (Radin,  1999). 

Barring  interpretation  issues,  full  implementation  is  not  instantaneous  (Reig, 
2000).  The  lag  between  enactment  and  implementation  is  a  topic  of  much  academic 
research.  Researchers  seeking  to  measure  the  effectiveness  of  reforms  vary  the 
implementation  lag  depending  on  the  analysis  (Holbrook,  2003;  Phillips,  2008; 
Cooper  M.  A.,  2002;  Drezner,  Jarvaise,  Hess,  Hough,  &  Norton,  1993).  One 
researcher  went  so  far  as  to  claim  the  actual  implementation  of  Congressional 
reform  relied  on  the  signing  of  subsequent  reform  acts  (Holbrook,  2003).  This 
research  used  two  1990s  acts  as  example,  the  reforms  included  in  the  Government 
Performance  and  Results  Act  (GPRA)  of  1993  and  the  Federal  Acquisition 
Streamlining  Act  (FASA)  of  1994  (Holbrook,  2003).  Argumentatively,  the  author 
states  the  FASA  of  1994  implemented  many  of  the  goals  of  GPRA  without  actually 
creating  many  new  policies.  The  primary  rationale  surrounding  this  measurement 
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delay  includes  aspects  of  inertia.  Many  of  the  policy  changes  within  a  reform  act 
strengthen  previously  enacted  reforms.  This  strengthening  solidifies  the  need  for 
true  implementation,  causing  understanding  and  appreciation  of  the  reform  by  the 
personnel  covered  within  the  reform  act  (Phillips,  2008]. 

Beyond  the  defining  of  full  policy  implementation,  the  agreement  on  actual 
effectiveness  measures  is  central  to  determining  the  success  of  acquisition  reforms. 
Previous  empirical  research  utilized  the  SAR  as  source  data  (Holbrook,  2003].  As 
previously  discussed  within  this  paper,  the  utilization  of  SAR  data  does  not 
immediately  preclude  the  legitimacy  of  resulting  interpretations;  however,  the 
information  must  be  thoroughly  analyzed  and  interpreted.  Beyond  the  utilization  of 
questionable  data  sources,  the  interpretation  of  cost  growth  lessens  the  applicability 
of  these  reports  (Hough,  1992].  Drezner  et  all.,  (1993]  and  Christensen,  Searle,  & 
Vickery,  (1999]  all  conducted  empirical  studies  with  results  showing  consistent 
annual  cost  growth  within  numerous  major  acquisition  programs.  The  impossibility 
of  measuring  the  cost  growth  of  a  weapon  system  had  acquisition  reforms  not  been 
implemented  versus  real  world  cost  growth  has  not  kept  researchers  from  trying. 
Should  acquisition  reforms  be  considered  ineffective  if  studies  show  no  decrease  in 
cost  growth  following  policy  implementation? 

Multiple  studies  into  the  effectiveness  of  acquisition  reforms  utilize  SAR  data 
against  a  reform  timeline  to  determine  if  any  correlation  exists.  Research  indicates 
that  aircraft  acquisitions  from  1960  through  1990  averaged  a  28%  cost  growth 
(Younossi  et  al.,  2007].  This  differs  from  reported  cost  growth  of  40%  for  programs 
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dating  1991  through  2001  [Phillips,  2008).  Examining  the  pre  and  post  reform  cost 
growth  percentages  research  indicates  that  there  is  no  statistical  difference  in 
growth  rate  [Holbrook,  2003).  A  similar  study  conducted  in  2004  by  Phillips 
supports  the  finding  of  no  statistical  difference.  One  positive  aspect  of  policy  reform 
centers  on  contract  management  cost  variance.  Research  indicates  a  positive 
correlation  between  contract  cost  variance  and  acquisition  policy  implementation 
[Holbrook,  2003).  While  a  correlation  adds  credence  to  arguments  supporting 
acquisition  reform,  it  does  not  prove  causality.  Contrarily,  the  possibility  remains 
that  the  increased  awareness  of  problems  decreases  cost  variance  as  much  as  the 
actual  reform  initiatives.  Additionally,  the  repetition  of  reform  themes,  such  as 
streamlining  or  leaning  the  process  or  Congressionally  stipulated  requirements, 
raises  concern  as  to  the  underlying  validity  of  new  reform  initiatives  [Phillips, 

2004). 

One  major  issue  with  previously  mentioned  studies  on  the  effectiveness  of 
acquisition  reform  efforts  centers  upon  the  selection  of  the  treatment  date.  In  the 
study  conducted  by  Phillips,  the  treatment  date  of  December  31, 1996,  differs  from 
other  research  using  a  December  31, 1991,  date  as  the  delineation  between  pre  and 
post  reform  implementation  [Smirnoff,  2006).  While  instituting  a  single  treatment 
date  simplifies  the  analysis  process,  the  results  require  further  investigation  and 
understanding  as  to  true  results.  The  reality  of  multiple  reform  initiatives  after  said 
treatment  dates  negates  the  certainty  of  the  results.  Accordingly,  a  study  conducted 


15 


in  2006  contradicts  much  of  the  previous  research  into  the  effectiveness  of 
acquisition  reforms. 

Rather  than  examining  acquisition  reform  initiatives  as  a  whole  along  a 
continuous  timeline,  Smirnoff  [2006]  examined  each  reform  individually.  Smirnoff 
identified  the  lack  of  variables  included  in  previous  research  as  a  potential 
weakness  in  the  research.  The  2006  thesis  utilized  a  fixed-effect  model  with  cost 
overruns  as  the  dependent  variable  and  multiple  independent  variables.  Smirnoff 
included  the  Packard  Commission,  Federal  Acquisition  Streamlining  Act,  Nunn- 
McCurdy  Act,  and  the  Defense  Acquisition  Workforce  Improvement  Act,  among 
other  non-reform  variables,  in  the  study.  Surprisingly,  Smirnoff  found  that  "the 
Packard  Commission  and  many  other  acquisition  reforms  would  have  reduced  cost 
overruns  had  the  other  factors,  such  as  decreasing  defense  budgets,  not 
overwhelmed  their  impact"  [Smirnoff,  2006].  The  results  indicated  that  variability 
of  numerous  aspects  of  major  defense  acquisition  programs  complicate  the  ability  to 
effectively  identify  exact  results  of  reform  initiatives. 

Smirnoff,  understanding  that  correlation  is  not  synonymous  with  causality, 
identified  areas  for  further  research,  seeking  to  identify  more  variables  needed  to 
create  a  stronger  model.  A  question  not  yet  researched  surrounds  the  placebo  effect 
identified  in  medical  studies.  Medical  research  takes  great  care  to  ensure 
participants  in  an  experiment  remain  unaware  of  who  is  receiving  actual  medication 
versus  those  receiving  a  placebo.  The  reasoning  behind  this  experiment  control  is 
the  reality  that  humans  act  differently  depending  on  what  they  perceive  to  be  the 
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truth.  Relating  the  placebo  effect  to  reform  initiatives,  does  the  idea  that  the 
government  is  looking  at  reforming  acquisition  have  as  much  an  effect  as  the  actual 
reform  initiative?  Could  a  placebo  reform,  which  actually  changes  nothing,  actually 
create  an  environment  where  cost  variance  improves? 

The  Cost-estimating  workforce 

The  research  into  acquisition  reform  effectiveness  traditionally  seeks  to 
identify  any  statistical  variance  in  cost  growth  pre/post  implementation  of  the 
reform.  Failing  to  identify  specific  areas  of  change  included  in  the  reform  is  an  area 
of  weakness  in  the  previous  studies.  For  example,  the  Packard  Commission 
recommended  among  other  items,  changing  the  organizational  culture  within  the 
acquisition  community.  Little  research  is  available  as  to  the  cost  growth 
implications  of  how  changing  the  organizational  culture  helped  reduce  cost  growth. 
The  research  naively  treats  all  reform  initiatives  generically,  no  matter  what  the 
actual  goals.  This  research  aims  at  changing  this  norm  and  creating  an 
understanding  of  what  specific  reactions  the  cost  community  implements  to  enact 
the  goals  of  acquisition  reform,  specifically  in  reaction  to  the  WSARA. 

The  John  Warner  National  Defense  Authorization  Act  of  2007  established  a 
goal  for  the  DoD  to  perform  cost  estimating  by  full-time  employees  or  members  of 
the  Armed  Forces.  As  a  result,  the  Air  Force  Under  Secretary  for  Acquisition 
directed  the  service  to  identify  current  capabilities  and  ensure  retention  of  the 
support  capabilities  currently  completed  by  inorganic  workers  (contractors].  The 
RAND  Corporation  conducted  a  census  of  the  cost-estimating  workforce  and 
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published  its  report  in  2009.  The  report  highlighted  many  areas  within  the  cost 
community  and  established  a  foundation  of  sorts  for  future  comparisons. 

The  census  indicated  that  in  2008  the  AF  cost-estimating  workforce, 
comprised  of  374  active  duty,  government  civilian,  and  private  contractors,  relied 
heavily  on  contractors  to  produce  initial  cost  estimates  (Vernez  &  Massey,  2009]. 
With  just  over  50%  of  the  workforce  being  organic  [military  and  civilian]  personnel, 
the  cost  estimating  community  was  dependent  on  the  private  contractor  for 
operational  success.  This  reality  raised  concern  for  the  ability  and  likelihood  of 
performing  cost  estimating  through  organic  means.  A  second  area  of  concern  was 
the  number  of  vacant  positions  reported  by  supervisors.  The  survey  indicated  that 
over  70  positions  throughout  the  cost  workforce  remained  vacant.  This  amounted 
to  over  16%  of  the  potential  cost-estimating  workforce  as  unfilled  billets.  WSARA 
drives  the  cost  community  toward  increasing  the  number  of  organic  workers,  yet 
how  likely  is  building  the  number  given  the  preexisting  abundance  of  vacant 
positions? 

Another  area  highlighted  by  the  survey  was  the  experience  of  the  cost 
analysts.  The  census  reported  that  approximately  51%  of  the  organic  workforce 
had  fewer  than  5-years'  experience  [Vernez  &  Massey,  2009].  Figure  1  below 
summarizes  the  information  reported  in  the  RAND  census.  Comparing  organic 
personnel  to  contractors  highlights  serious  differences  in  experience  levels. 
Contractor  experience  remains  largely  consistent  as  a  percentage  for  each 
experience  range.  However,  organic  personnel  skew  largely  to  the  right  with 
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approximately  70%  of  personnel  having  10  years  or  less  experience.  Additionally, 
the  percentage  of  organic  personnel  having  greater  than  20  years’  experience  is 


dramatically  less  than  the  contractor  personnel. 


60% 


^  50% 

o 

c 

c 

®  40% 

o 

a 


o 

o 

W) 

s 

c 

o 

u 

u 

o 

On 


30% 


20% 


10% 


0% 


<5 


RAND  Census 


I  Organic 
I  Contractor 


6-10  11-20 

Years  of  Experience 


>20 


Figure  1:  Experience  as  Reported  in  the  RAND  Census 

Specific  cost-estimating  certification  was  a  second  area  of  experience  falling 
into  an  area  of  grave  concern.  The  report  stated  that  over  two-thirds  of  the  entire 
workforce  lacked  a  cost-estimating  certification.  Between  the  numerous  vacancies, 
the  inexperience  of  the  workforce,  and  the  lack  of  certification,  the  cost  estimating 
community  was  in  a  precarious  position. 

An  area  lacking  study  within  the  cost  community  is  the  way  the  current 
organizational  structure  affects  efficiency.  A  great  deal  of  academic  research  within 
the  management  community  seeks  to  understand  how  various  elements  of 
interaction  affect  organizations. 
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Published  research  on  leadership,  motivation,  leader  member  exchange, 
trust,  and  organizational  commitment,  among  many  other  variables,  ah  identify  how 
companies  might  tailor  and  deliver  information  on  policy  changes  to  achieve  the 
greatest  support  from  workers.  One  developing  area  in  research  is  the  aspect  that 
all  companies  rely  to  varying  degrees  on  technical  support  for  communication  and 
production.  As  such,  the  virtualness  of  a  company  affects  many  areas  of  production 
and  efficiency. 

Virtualness 

Virtualness  embodies  multiple  factors  that  create  physical  and  psychological 
distance  between  members  of  an  organization.  Current  literature  identifies  up  to 
four  primary  dimensions,  which  influence  the  level  of  virtualness  (Hertel  et  al., 

2005;  Erskine,  2007;  Griffith  et  al.,  2003;  Fiol  &  O'Connor,  2005).  One  dimension  of 
virtualness  is  team  distance  or  separation.  However,  this  distance  is  a  complex 
variable  in  that  it  not  only  embodies  just  relative  proximity  of  team  members; 
distance  also  includes  a  mutual  understanding  of  work  environment  and 
hierarchical  separation  [Erskine,  2007).  A  second  dimension  of  virtualness  is  the 
reliance  on  technical  support  [Kirkman  &  Mathieu,  2005;  Bell  &  Kozlowski,  2002; 
Griffith  et  al.,  2003).  Technical  support  includes  multiple  aspects,  such  as  the  use  of 
electronic  tools  for  communicating  and  work  production.  A  third  dimension  of 
virtualness  is  the  value  of  information  provided  by  technology,  henceforth  known  as 
technological  richness  [Carlson  &  Zmud,  1999;  Griffith  et  al.,  2003).  A 
differentiation  between  the  use  of  technology  and  the  value  added  by  said 
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technology  allows  for  a  more  complete  analysis  of  technological  utilization  in  team 
environments.  The  fourth  dimension  is  synchronicity  of  the  team  members 
(Kirkman  &  Mathieu,  2005;  Hertel  et  al.,  2005].  Within  the  context  of  this  paper,  a 
team  includes  structured  teams  and  also  supervisor  subordinate  relations.  The 
synchronicity  among  team  members  may  vary  from  the  synchronous 
communication  found  in  face-to-face  interactions  to  the  asynchronous  interactions 
in  voicemail  or  email  messages.  Having  introduced  the  four  dimensions  of 
virtualness,  1  will  spend  a  few  pages  developing  a  greater  knowledge  of  each. 

Distance 

The  aspect  of  distance  inherent  in  the  virtualness  of  a  team,  in  its  most 
simplistic  form,  is  identifiable  as  physical  proximity  between  members.  Elementary 
measurements  for  distance  limit  understanding  to  mileage  between  members. 
However,  a  more  thorough  understanding  of  distance  is  required  to  appreciate  the 
level  of  virtualness  of  a  team.  Physical  proximity  is  but  one  aspect  of  distance, 
which  creates  separation  of  team  members.  Mutual  understanding  and  agreement 
on  work  environment  is  a  second  aspect  of  distance.  The  cost  analysis  community 
of  the  AF  presents  an  excellent  example  of  how  limiting  distance  to  physical 
proximity  fails  to  capture  the  complexity  of  distance.  AF  organizations  often 
collocate  teams  of  active  duty  personnel,  civilian  government  service  employees, 
and  contractors.  The  proximity  of  these  teams  may  be  measured  in  feet;  however, 
the  team  members  understanding  of  work  pressures  differ  greatly.  The  different 
types  of  personnel  all  experience  differing  forms  of  work  stress  and  production 
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goals.  As  such,  a  more  complete  understanding  of  distance  must  include  non¬ 
tangible  aspects  of  distance. 

Technical  Support 

Reliance  on  technical  support  does  not  in  itself  determine  a  level  of 
virtualness.  Researchers  argue  that  with  regards  to  team  or  personal  interaction, 
technical  support  augments  the  ability  to  convey  information.  A  geographically 
dispersed  team  may  be  heavily  dependent  on  technology  to  facilitate  the  progress  of 
team  goals  (Erskine,  2007].  Argumentatively,  a  geographically  dispersed  team  may 
not  use  technology  other  than  to  communicate  the  time  and  location  for  the  next 
team  meeting.  Since  technological  dependence  may  vary  independent  of  physical 
location  of  team  members,  it  cannot  be  a  proxy  for  distance.  As  such,  dependence 
on  technology  is  a  separate  dimension  of  virtualness. 

Richness  of  Technology 

Communication  is  vital  to  the  success  of  a  team  (Carlson  &  Zmud,  1999]. 
Technology  allows  a  team  to  conduct  many  operations  that  previously  required 
close  proximity.  However,  the  value  of  technology  is  dependent  on  the  richness  of 
the  information  relayed.  If  a  team  communicates  through  electronic  mail,  but  the 
receiver  of  the  message  misunderstands  the  intent  of  the  sender,  then  there  is  no 
value  in  the  technology.  Additionally,  a  great  deal  of  communication  is  non-verbal, 
which  may  be  lost  in  technology.  In  order  to  counteract  the  possible  deterioration 
of  richness,  a  team  must  ensure  that  the  technology  allows  sufficient  flexibility  to 
meet  all  needs  (Carlson  &  Zmud,  1999]. 
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Synchronicity 


The  literature  surrounding  synchronicity  indicates  that  communications 
conducted  in  real  time  are  synchronous.  Instituting  a  delay  into  a  communication 
chain  creates  an  asynchronous  exchange.  A  great  deal  of  research  into  the 
implications  of  synchronicity  of  communication  involves  different  aspects  of  the 
exchanging  of  information  (Kirkman  &  Mathieu,  2005].  Early  researchers 
hypothesized  that  asynchronous  relationships  degraded  the  value  of  the 
communications.  However,  further  research  highlighted  added  value  to 
asynchronous  communications,  which  counteracts  the  degradation.  Asynchronous 
communication  allows  for  the  tailoring  of  the  message  to  convey  the  desired 
meaning  (Kirkman  &  Mathieu,  2005].  1  must  stress,  synchronicity  is  not  dependent 
specifically  on  the  technology  used  to  communicate.  A  computer  may  provide  both 
synchronous  communication  in  the  form  of  instant  messaging  and  asynchronous 
communication  via  email.  It  is  counterintuitive  that  in  the  context  of  synchronicity, 
the  almost  archaic  method  of  a  handwritten  and  post  mailed  letter  is  more  virtual 
than  a  cell  phone  call. 
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III.  Methodology 


Chapter  II  provided  information  on  the  development  of  acquisition  reform, 
studies  on  the  effectiveness  of  reform,  and  the  current  hotbed  that  is  virtualness. 
Chapter  III  introduces  the  methodology  utilized  to  access  virtualness  and  garners  a 
greater  understanding  of  the  current  work  habits  of  the  cost  community.  The 
methodology  includes  the  rationale  behind  employing  and  developing  a 
questionnaire.  The  discussion  on  the  questionnaire  contained  within  this  chapter 
addresses  the  questionnaire  itself;  the  collected  results  and  analysis  appear  in 
Chapter  IV. 

Methodology 

A  meta-analysis  of  selected  acquisition  reports  or  previous  studies  fails  to 
achieve  a  complete  understanding  of  the  atmosphere  present  within  the  cost 
community.  The  few  studies  available  for  analysis  fail  to  achieve  the  depth 
necessary  to  present  greater  than  a  cursory  explanation  of  time  allocation.  The 
hypotheses  studied  within  this  paper  were  tested  utilizing  data  acquired  through  an 
online  questionnaire.  The  data  sample  of  the  cost  analysis  community,  which 
includes  approximately  400  individuals,  333  of  which  personnel  received  the 
invitation  to  complete  the  questionnaire.  The  cost  acquisition  community  consists 
of  four  primary  headquarter  organizations  and  three  operating  locations. 

The  dispersion  of  the  personnel  presents  excellent  applicability  to 
developing  the  virtualness  measure.  The  four  primary  locations  include  Hancsom 
AFB  located  in  Boston,  Massachusetts,  Air  Force  Cost  Analysis  Agency  located  in 
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Washington,  D.C.,  Wright  Patterson  AFB  located  in  Dayton,  Ohio,  and  Los  Angeles 
AFB  located  in  Los  Angeles,  California.  The  three  operating  locations  include 
Oklahoma  City  Air  Logistics  Center,  Ogden  Air  Logistics  Center,  and  Warner-Robins 
Air  Logistics  Center.  Beyond  the  physical  separation  of  the  location,  the  military 
environment  introduces  hierarchical  and  cultural  discontinuities  between 
personnel. 

Measures 

The  questionnaire  requested  individuals  to  rate  appropriately  utilizing  a  five- 
point  Likert-type  scale  from  1  (strongly  disagree)  to  5  (strongly  agree)  unless 
otherwise  noted.  This  study  adapted  existing  measures  of  proven  reliability  and 
accuracy.  Appendix  D  presents  the  questionnaire  in  document  form. 

Questionnaire 

This  study  employed  two  similar  questionnaires  in  order  to  accurately  assess 
the  cultural  nuances  contained  with  the  cost  analysis  community.  Multiple  benefits 
arose  from  the  necessity  of  tailoring  a  separate  questionnaire  for  the  supervisor  and 
the  subordinate.  The  two  questionnaires  allowed  for  a  matched-pair  comparison  of 
measures,  ultimately  ensuring  reliability  of  the  created  virtualness  measure. 

Chapter  4  of  this  paper  includes  more  specific  discussion  surrounding  the  results  of 
the  matched-pair  study.  Limitations  encountered  during  the  participant  selection 
phase  excluded  pre-determining  which  participants  supervised  individuals.  As 
such,  the  web-based  survey  utilized  a  branched  design  in  which  the  participant 
selected  the  applicable  questionnaire.  Participants  indicated  supervisory 


25 


responsibilities  in  question  1  of  the  survey;  as  a  result,  those  individuals  who 
indicated  responsibility  for  supervising  others  completed  the  questionnaire 
targeting  supervisors.  The  description  of  individual  measurement  items  explains 
the  specific  adaptations  for  supervisor  or  non-supervisor  questionnaire. 

Trust 

The  trust  measures  for  non-supervisor  participants  are  an  adaptation  of 
institutionally  accepted  questions  as  tested  by  Ballinger,  Schoorman,  and  Lehman 
(2009].  Cognitive-based  trust  items  capture  the  perception  of  the  subordinate 
regarding  the  established  track  record  of  the  supervisor.  Affect-based  trust 
measures  seek  an  understanding  of  the  relationship  between  supervisor  and 
subordinate.  Affect-based  trust  is  less  about  the  actual  performance  and  more 
about  likability  or  fondness.  The  non-supervisor  questionnaire  utilizes  seven  items 
specifically  addressing  trust  towards  the  supervisor.  One  example  item  is,  "My 
supervisor  keeps  my  interests  in  mind  when  making  decisions." 

Trust  measures  for  participants  who  indicated  supervisory  responsibilities 
utilize  five  of  the  same  items  as  the  non-supervisory  questionnaire.  Adapting  the 
items  for  supervisors  primarily  involved  replacing  the  term  supervisor  with 
subordinate.  A  sample  item  for  the  supervisor  is,  "It  is  important  for  me  to  have  a 
good  way  to  keep  an  eye  on  my  subordinate.”  In  order  to  keep  the  completion  time 
of  the  questionnaire  manageable,  the  supervisor  portion  only  uses  five  of  the  seven 
trust  items. 
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Leadership  Member  Exchange  (LMX) 

The  inclusion  of  LMX  hinges  on  recent  literature  indicating  a  correlation 
between  virtualness  and  LMX  (Erskine,  2007;  Howell  &  Hall-Merenda,  1999]. 
Virtualness  as  a  moderator  of  LMX  indicates  that  strong  leadership  may  overcome 
the  negative  effects  of  virtualness  [Kirkman,  Rosen,  Tesluk,  &  Gibson,  2004].  The 
questionnaire  utilizes  eight  items  adapted  from  Liden,  Wayne,  and  Stilwell  [1993]. 

A  sample  measure  is,  "1  know  where  1  stand  with  my  supervisor.” 

Job  Satisfaction 

Job  satisfaction  measures  appear  on  the  non-supervisor  questionnaire  while 
absent  from  the  supervisor  questionnaire.  Including  four  items  adapted  from  Weiss, 
Nicholas,  and  Daus  [1999]  allowed  for  greater  control  on  variables  modified  by 
virtualness.  [1999].  A  sample  measure  for  job  satisfaction  is,  "All  in  all  1  am 
satisfied  with  my  job.” 

Turnover  Intentions 

Turnover  intention  items  stem  from  an  adaptation  of  Wayne,  Shore,  and 
Liden  [1997].  The  supervisor  questionnaire  excludes  turnover  intentions  as  a 
measurement  item.  A  sample  question  for  turnover  intentions  is,  "1  am  seriously 
thinking  about  quitting  my  job.”  The  questionnaire  includes  five  items  specifically 
tailored  to  measure  turnover  intention. 

Organizational  Commitment 
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Organizational  commitment  items  included  in  the  non-supervisor 
questionnaire  originate  from  the  1997  book  Commitment  in  the  Workplace  [Meyer  & 
Allen].  The  questionnaire  utilizes  eight  items  in  the  assessment  of  non-supervisors’ 
commitment  to  the  organization.  A  sample  item  is,  "1  would  be  happy  to  spend  the 
rest  of  my  career  with  this  organization." 

Leader  Appraisal  of  the  Member's  Performance 

The  supervisor  questionnaire  utilizes  four  items  adapted  from  the  Liden  et  al. 
study  of  leader  appraisal  of  the  member's  performance  [Howell  &  Hall-Merenda, 
1999].  Supervisors  completed  the  measures  for  each  military  and  government 
civilian  subordinate.  A  sample  measure  is,  "This  subordinate  is  superior  to  other 
subordinates  1  have  supervised  before." 

Virtualness  (adapted  from  (Carlson  &  Zmud,  1999) 

The  virtualness  portion  of  the  questionnaire  differs  from  other  measures  in 
that  the  underlying  goal  is  to  develop  a  universal  measure.  Scholars  dispute  exactly 
which  dimensions  directly  measure  virtualness.  The  questionnaire  seeks  to  gain  a 
further  understanding  of  virtualness  by  developing  an  accurate  measure  of 
assessing  all  dimensions.  In  order  to  create  this  measure,  the  questionnaire 
contains  a  vastly  increased  number  of  questions  in  each  of  the  four  dimensions  of 
virtualness.  My  goal  in  this  effort  is  to  differentiate  between  each  of  the  four 
underlying  dimensions  of  virtualness  and  develop  a  measure  for  each. 
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Distance 


1.  My  supervisor  thoroughly  understands  my  working  environment. 

2.  My  supervisor  works  within  close  physical  proximity  to  me. 

3.  My  team  members  understand  my  job  requirements. 

4.  1  really  understand  why  people  behave  the  way  they  do  in  my  organization. 

5.  1  often  interact  with  team  members  not  co-located  with  me. 

6.  1  have  a  good  understanding  of  the  environment  in  which  my  organization 
operates. 

7.  Time  zones  add  difficulty  to  my  work. 

8.  My  supervisor  understands  the  daily  requirements  of  my  job. 

9.  My  supervisor  and  1  have  a  common  understanding  of  work  requirements. 

10. 1  have  a  good  sense  of  the  dynamics  within  my  organization. 

11. 1  know  what  other  people  in  my  organization  are  doing. 

Technical  Support 

12.  My  supervisor  carries  an  email-enabled  smartphone  such  as  a  Blackberry™. 

13. 1  am  unable  to  communicate  with  my  team/supervisor  without  electronics,: 
telephone,  computer,  etc. 

14.  My  supervisor  is  available  on  a  cell  phone  throughout  the  workday. 

15. 1  primarily  communicate  with  my  team  members  through  email. 

16. 1  use  email  to  communicate  with  my  supervisor. 

17. 1  primarily  complete  my  work  through  computers. 
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18. 1  contact  my  supervisor  through  telephone  conversations. 

19.  Without  computers  I  would  be  unable  to  accomplish  my  job. 

20. 1  utilize  information  technology  in  my  daily  interactions  with  my  supervisor. 

21.  My  supervisor  carries  a  cell  phone. 

Richness  of  technology 

22. 1  often  speak  in  person  with  my  supervisor  to  clarify  messages  received  through 
electronic  formats. 

23.  Utilizing  email  makes  it  difficult  to  understand  the  tone  of  messages  from  my 
supervisor. 

24. 1  am  easily  able  to  understand  a  variety  of  different  cues  [e.g.,  emotional  tone, 
feelings]  from  my  supervisor. 

25. 1  often  seek  instructions  sent  via  email  from  my  supervisor  regarding  work 
requirements. 

26. 1  am  easily  able  to  understand  the  message  from  my  supervisor. 

27. 1  am  easily  able  to  tailor  my  messages  to  my  supervisor. 

28. 1  often  speak  in  person  with  my  coworkers  to  clarify  messages  received  through 
electronic  formats. 

29. 1  am  able  to  use  rich  and  varied  language  when  communicating  with  my 
supervisor. 

30.  It  is  easy  to  exchange  timely  feedback  with  my  supervisor. 

31. 1  am  easily  able  to  maintain  multiple  conversations  with  co¬ 
workers/supervisors. 


30 


Synchronicity 


32.  It  negatively  affects  my  work  when  my  supervisor  is  absent. 

33.  My  supervisor  is  available  whenever  I  need  him/her. 

34.  My  supervisor's  work  schedule  is  in-synch  with  my  own  work  schedule. 

35.  My  supervisor  answers  my  questions  on  the  same  day  I  send  the  email. 

36.  It  is  often  difficult  to  get  in  touch  with  my  supervisor. 

37.  My  supervisor  and  I  often  have  misunderstandings  driven  by  the  differences  in 
our  schedules. 

38.  My  supervisor  and  I  have  difficulties  aligning  our  schedules. 

39.  My  schedule  changes  are  based  on  my  supervisor’s  schedule. 

40.  My  supervisor  and  I  always  seem  to  be  in  tune  as  to  what  we  are  doing. 

41.  My  supervisor  responds  to  my  messages  [e.g.,  phone,  email]  in  a  timely  manner. 
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IV.  Analysis  and  Results 


Chapter  Overview 

As  previously  discussed,  the  cost  analysis  community  endures  a  myriad  of 
accusations  surrounding  the  seemingly  endless  growth  in  cost  for  DoD  major 
weapon  acquisitions.  As  such,  1  hypothesized  that  there  is  a  correlation  between  the 
organizational  structure  of  the  cost  analysis  community  and  virtualness. 
Furthermore,  the  level  of  virtualness  apparent  in  the  cost  community  impacts  the 
ability  to  produce  accurate,  efficient  cost  estimates.  Table  3  below  summarizes  the 
hypothesized  relationships  between  virtualness  and  various  facets  of  individual 
behavior. 


Table  3:  Hypothesized  Relationships  of  Virtualness 


Summary  of  expected  correlations 

Factor 

Independent  Variable 

Relationship 

Virtualness 

Trust 

- 

Virtualness 

Job  Satifaction 

- 

Virtualness 

Orgizational  Commitment 

- 

Analytical  Approach 

As  accepted  definitions  and  measures  for  virtualness  vary  within  the 
academic  discipline,  this  research  aims  at  creating  a  reliable  measure.  The  initial 
creation  of  the  measure  adapts  accepted  individual  dimension  measures  where 
applicable,  as  in  the  case  of  media  richness  [Carlson  &  Zmud,  1999].  The 
questionnaire  includes  41  individual  measures  for  virtualness  divided  among  the 
four  dimensions.  All  dimensions,  except  for  distance,  contain  10  questions  with 
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distance  using  11.  Distance  is  comprised  of  two  factors,  physical  distance  and 
psychological  distance,  thus  requiring  an  additional  question.  Ultimately,  the 
measure  uses  only  16  questions  in  hopes  of  simplifying  the  data  gathering  process 
in  future  questionnaires.  While  the  structure  of  questionnaire  allowed  for  a 
matched-pair  comparison  between  data  sets,  the  limited  sample  set  precludes  this 
use.  Alternatively,  1  employ  structural  equation  modeling  to  analyze  the  results  and 
create  a  measure  of  virtualness. 

Utilizing  reliability  measures  as  well  as  face  validity  and  statistical  analysis, 
results  in  the  reduction  of  the  41  questions  to  approximately  four  questions  per 
dimension.  Once  reducing  the  individual  measures  to  approximately  16  questions,  1 
utilized  structural  equation  modeling  to  confirm  the  relationships.  1  am  examining 
how  the  four  dimensions  of  virtualness  combine  into  a  measure  of  virtualness.  The 
measure  allows  for  further  exploration  into  how  virtualness  effects  trust, 
organizational  commitment,  and  job  satisfaction.  The  correlation  between  the 
independent  variables  and  the  detrimental  variable  of  turnover  intention  should  not 
be  understated.  Figure  2  below  depicts  the  anticipated  model  and  effects  of  the 
independent  variable  virtualness  and  turnover  intentions.  However,  in  addition  to 
the  anticipated  relationship  between  virtualness  and  turnover  intention,  a 
qualitative  assessment  of  the  gathered  results  is  necessary  for  a  complete 
understanding  of  the  unquantifiable  steps  taken  to  counteract  the  negative  views  of 
the  cost  community. 
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Figure  2:  Relationship  Diagram  from  Virtualness  to  Turnover  Intention 

Results 

Cost  Analysis  Community 

The  purpose  of  this  section  is  to  report  the  results  of  the  information 
gathered  from  both  the  supervisor  and  non-supervisor  questionnaires.  The 
supervisor  questionnaire  contains  many  qualitative  questions  seeking  to  gain  an 
understanding  of  the  current  condition  of  the  cost  estimating  community.  This 
section  summarizes  and  when  applicable  compares  the  results  to  the  RAND  study 
(Vernez  &  Massey,  2009).  Primarily,  the  questionnaire  focuses  on  one  underlying 
aspect,  time  utilization. 


Time  Usage 

Similar  to  the  RAND  study,  my  questionnaire  asks  respondents  to  indicate 
how  they  spent  their  work  time  for  the  last  six  months.  While  this  creates  an 
estimate  of  historical  time  utilization  in  a  single  point,  comparing  the  results  against 
the  RAND  study  allows  for  a  quasi-longitudinal  view.  1  must  state  that  while  the 
RAND  labels  its  study  as  a  census  and  includes  contractors,  the  questionnaire 
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completed  for  this  thesis  sampled  only  military  and  civilian  cost  estimators.  The 
census  raises  a  reasonable  point  when  stating  that  as  cost  estimating  requirements 
change  based  on  many  external  factors  (program  age,  funding  amounts  and 
requirements,  congressional  oversight,  etc.],  averaging  the  responses  allows  for  a 
close  approximation  of  general  cost  analysis  time  utilization  (Vernez  &  Massey, 
2009].  Table  4  below  reports  the  time  usage  findings  of  the  RAND  census. 

Table  4:  Time  spent  on  cost  analysis  (Vernez  &  Massey,  2009) 


Time  Spent  Doing  Cost  Estimation  Over  Previous  Six  Months,  by  Personnel  Type,  2008  (%) 


Personnel  Type 

None 

Less 

than  25 

25  to 

49.9 

50  to 

74.9 

75  to 

99.9 

100 

Total 

Organic  (n=184) 

5 

17 

17 

12 

14 

35 

100 

Contractor  (n=190) 

1 

10 

2 

8 

47 

32 

100 

All  (n=374) 

3 

13 

9 

10 

32 

33 

100 

SOURCE:  RAND  census,  2008. 

NOTE:  Percentages  may  not  add  to  100  because  of  rounding. 


Table  4  above  shows  time  spent  on  cost  estimating;  however,  the  RAND 
study  fails  to  identify  what  tasks  it  considers  cost  estimating.  The  questionnaire 
designed  for  this  thesis  includes  a  matrix-like  breakout  of  14  tasks.  1  developed 
these  tasks  through  preliminary  interviews  with  current  cost  estimators.  Table  5 
below  shows  the  matrix  included  in  the  cost  questionnaire. 
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Table  5:  Non-Supervisor  Time  Matrix  Questionnaire 


Within  the  last  6  months,  what  %  time  spent  on  following  activities: 

%  Time 

a.  %  time  spent  analyzing  Earned  Value  Management  (EVM)  data 
(i.e.,  CPR,  CSSR)  using  Winsight  or  other  program  in  support  of 
program  management 

b.  %  time  spent  using  Earned  Value  (EV)  data  in  support  of  creating 
estimates 

c.  %  time  spent  doing  non-cost-related  financial  management 

d.  %  time  spent  managing  support  contractors 

e.  %  time  consulting  with  PM  or  personnel  in  other  departments  to 
discuss  and  formulate  estimates  or  resolve  issues 

f.  %  time  preparing  initial  estimates 

g.  %  time  reconciling  estimates  with  either  AFCAA  or  OSD 

h.  %  time  reviewing  estimates 

i.  %  time  completing  post  estimate  documentation 

j.  %  time  completing  what-if  drills  for  PM  due  to  program  changes 

or  budget  constraints 

k.  %  time  spent  in  meetings 

1.  %  time  training  (relating  to  cost  only;  include  both  time  spent  in 

training  and  time  spent  training  others) 

m.  %  time  spent  in  non-cost-related  training  (annual  training  - 
Information  Protection,  Human  Trafficking,  etc;  do  not  include  time 
spent  doing  cost-related  training) 

n.  %  time  Other 

Total;  Must  add  up  to  100% 

0% 

These  14  items  include  tasks  directly  related  to  cost  estimation  as  well  as 
general  work  activities.  In  order  to  compare  the  responses  gathered  from  the 
questionnaire  against  the  RAND  census,  1  have  assigned  items  a,  b,  e,  f,  g,  h,  i,  and  j  to 
cost  estimating  activities.  1  excluded  item  k  from  the  comparison  of  time  utilization, 
since  may  include  both  cost-related  and  non-cost-related  meetings.  Table  6  below 
compares  the  RAND  findings  against  the  time  utilization  of  organic  personnel 
reported  through  the  questionnaire. 
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Table  6:  Time  Usage  Comparison  Between  RAND  and  Research 


Time  Spent  Doing  Cost  Estimation  Over  Previous  Six  Months 

None 

< 

25% 

25  to 

49.9% 

50  to 

74.9% 

75  to 

99.9% 

100% 

Total 

RAND  * 

5% 

17% 

17% 

12% 

14% 

35% 

100% 

Thesis  Study 

2% 

3% 

14% 

42% 

38% 

0% 

100% 

Thesis  Study  ** 

1% 

2% 

7% 

21% 

68% 

1% 

100% 

*SOURCE:  RAND  census,  2008. 

**  Indicates  "Cost  estimating  time  plus  "time  spent  in  meetings" 
NOTE:  Percentages  may  not  add  to  100  because  of  rounding 

The  questionnaire  results  highlight  a  major  flaw  in  the  RAND  study.  RAND 
indicates  approximately  35%  of  organic  personnel  spend  100%  of  their  time  on  cost 
estimating  tasks.  Perhaps  the  nature  of  the  RAND  study  differs  from  the 
questionnaire;  nevertheless,  my  findings  differ  greatly.  By  coding  the  previously 
mentioned  questionnaire  items  as  cost  estimating  tasks,  no  respondents  indicate 
they  spend  100%  of  their  time  on  cost  estimation.  Also  of  significance  is  the  percent 
of  personnel  who  report  spending  less  than  25%  of  their  time  on  cost  estimating 
tasks.  The  questionnaire  presented  14  different  activities  for  time  allocation  and 
only  5%  of  respondents  indicated  they  spent  less  than  25%  of  their  time  doing  cost¬ 
estimating  activities.  For  completeness,  the  third  row  in  Table  4  shows  the 
percentage  of  time  utilization  when  including  "time  spent  in  meetings.”  This  skews 
the  data  and  shows  a  much  higher  percentage  of  personnel  spending  over  75%  of 
their  time  in  cost  estimating  activities. 

Figure  3  below  graphically  compares  the  time  utilization  per  category  for 
military  and  civilians  as  reported  by  the  employee.  The  primary  goal  of  the  time 
utilization  portion  of  the  questionnaire  is  to  identify  how  the  analyst  spends  time. 
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As  such,  asking  supervisors  to  report  their  time  utilization  failed  to  achieve  the 
desired  result.  Instead,  the  questionnaire  instructed  supervisors  to  breakout  how 
their  cost  analysis  team  utilizes  its  time. 


Employee  view  of  time  utilization 


20% 
^  18% 
I  16% 

S  14% 

I  12% 
f  10% 
■2  8% 

I  6% 

^  4% 

^  2% 
0% 


■  Civilian 

■  Military 


Figure  3:  Aggregate  of  Time  Usage  as  Reported  by  Employee 

While  Table  5  above  shows  the  questionnaire  matrix  presented  to  the  non- 
supervisory  personnel.  Table  7  below  shows  the  matrix  completed  by  supervisors. 
The  questionnaire  asks  supervisors  to  indicate  how  many  employees  for  each  of  the 
four  categories  they  supervise.  The  supervisor  then  indicates  what  percentage  of 
time  each  personnel  category  spends  on  the  14  individual  tasks.  The  14  individual 
tasks  are  consistent  with  those  presented  to  non-supervisory  personnel.  The 
primary  difference  of  viewpoint  creates  an  excellent  opportunity  to  identify  areas  of 
differing  opinion. 
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Table  7:  Supervisor  Time  Matrix  Questionnaire 


Number  employed  of  each 

Civilian 

Military 

Contractor 

FFRDC 

Within  the  last  6  months,  what  %Time  spent  on  following  activities: 

%Time 

%  Time 

%  Time 

%Time 

a.  %  time  spent  analyzing  Earned  Value  Management  (EVM)  data  (i.e.  CPR,  CSSR) 
using  winsight  or  other  program  in  support  of  program  management 

b.  %  time  spent  using  Earned  Value  (EV)  data  in  support  of  creating  estimates 

c.  %  time  spent  doing  non-cost  related  financial  management 

d.  %  time  spent  managing  support  contractors 

e.  %  time  consulting  with  PM  or  personnel  in  other  departments  to  discuss  and 
formulate  estimates  or  resolve  issues 

f.  %  time  preparing  initial  estimates 

g.  %  time  reconciling  estimates  with  either  AFCAA  or  OSD 

h.  %  time  reviewing  estimates 

i.  %  time  completing  post  estimate  documentation 

j.  %  time  completing  what-if  drills  for  PM  due  to  program  changes  or  budget 

k.  %  time  spent  in  meetings 

1.  %  time  training  (relating  to  cost  only:  included  both  time  spent  in  training  and 

time  spent  training  others) 

m.  %  time  spent  in  non-cost  related  training  (annual  training  -  Information 
Protection,  Human  Trafficking,  etc;  do  not  include  time  spent  doing  cost  related 

n.  %  time  Other 

Total:  Must  add  up  to  100% 

0% 

0% 

0% 

0% 

While  only  19  supervisors  completed  this  portion  of  the  questionnaire,  they 
indicated  they  supervise  a  total  of  122  organic  and  59  contractor  personnel.  As 
such,  they  represent  a  significant  sample  of  the  cost  estimating  supervisory 
personnel.  Table  8  below  shows  the  mean  totals  for  each  of  the  14  time  categories 
broken  out  by  employee  versus  supervisor  and  civilian,  military,  and  contractor. 
The  table  requires  the  number  of  personnel  within  each  employment  category 
above  the  %  Time  columns.  The  total  number  of  employees  reporting  in  the  survey 
is  105,  and  the  supervisors  report  on  a  total  of  181  individuals,  including 
contractors. 

Figure  4  below  shows  the  reported  time  usage  for  the  cost  analysis  team 
viewed  from  the  supervisors’  standpoint.  As  the  figure  indicates,  supervisors 
estimate  that  contractors  spend  considerably  more  time  than  organic  personnel  on 
the  task  of  actual  initial  estimate  creation.  Understandably,  supervisors  indicate 
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that  contractors  spend  less  time  on  training  and  managing  other  support  contracts, 
Interestingly,  supervisors  returned  information  indicating  that,  for  the  most  part, 
military  and  civilians  spend  similar  percentages  of  time  for  most  tasks.  One  large 
note  of  difference  is  with  supervisors  indicating  military  spend  considerably  more 
time  in  the  area  of  non-cost-related  FM  activities. 


Figure  4:  Aggregate  of  Time  Usage  as  Reported  by  Supervisor 

While  the  comparison  against  the  RAND  census  highlights  some  interesting 
aspects  of  reported  time  utilization,  comparing  the  non-supervisors  versus 
supervisors’  responses  allows  for  an  alternative  assessment.  Question  1  of  the 
survey  asks  the  respondent  if  he  or  she  supervises  personnel.  Based  on  the  answer, 
supervisors  completed  a  similar  but  more  qualitative  questionnaire.  One  of  the 
supervisor-specific  aspects  of  the  questionnaire  is  to  indicate  the  number  of 
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personnel  supervised  (broken  into  four  categories:  military,  civilian,  contractor,  and 
Federally  Funded  Research  and  Development  Centers  -  FFRDC]  and  to  indicate  how 
much  time  individuals  in  each  category  spend  on  the  14  tasks.  Table  8  below 
summarizes  the  mean  percent  reported  per  category  as  well  as  personnel  group. 
The  lack  of  FFRDC  under  supervisors  is  due  to  the  lack  of  data.  Not  a  single 
supervisor  indicated  his  or  her  cost  team  contained  FFRDC  personnel.  Comparing 
the  mean  time  per  category  within  personnel  groups  shows  that  for  the  most  part 
supervisors  estimate  the  time  utilization  with  much  the  same  breakdown  as 
employees.  As  mentioned  earlier,  the  largest  single  mean  percentage  of  time  is  with 
contractors  preparing  initial  estimates  at  an  incredible  25%. 


Table  8:  Aggregate  Time  Usage  Matrix 


Employees 

Supervisors 

11  94 

109 

13 

59 

Civilian 

Military 

Civilian 

Military 

KTR 

Analyzing  EVM  Data 

3% 

3% 

7% 

13% 

16% 

Using  Earned  Value  (EV)  data 

3% 

4% 

2% 

3% 

6% 

Non-cost-related  FM 

7% 

4% 

4% 

11% 

2% 

Managing  support  contractors 

3% 

3% 

5% 

4% 

1% 

Consulting  with  PM 

13% 

13% 

11% 

9% 

18% 

Preparing  initial  estimates 

18% 

13% 

11% 

11% 

25% 

Reconciling  estimates 

5% 

8% 

2% 

1% 

4% 

Reviewing  estimates 

12% 

10% 

8% 

4% 

3% 

Post-estimate  documentation 

6% 

5% 

5% 

8% 

7% 

What-if  drills 

5% 

6% 

6% 

3% 

5% 

Meetings 

12% 

10% 

12% 

15% 

11% 

Cost-related  training 

6% 

6% 

9% 

11% 

1% 

Non-cost-related  training 

4% 

3% 

4% 

9% 

1% 

Other 

4% 

13% 

14% 

0% 

2% 

Total 

100% 

100% 

100% 

100% 

100% 
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Figure  5  below  compares  the  mean  time  reported  by  military  employees 
against  the  estimated  percentage  breakout  as  reported  by  supervisors.  The 
supervisors  indicated  a  very  different  time  allocation  when  compared  to  the 
military  employee’s  time  usage.  Average  times  on  the  linear  graph  do  not  seem  to 
follow  or  mirror  each  other. 


Figure  5:  Time  Allocation  Comparison  for  Military  Against  Supervisor  Expectation 

Figure  6  below  contains  the  same  information  as  Figure  4  above  except  it 
compares  civilian  employees'  reported  time  utilization  against  the  supervisors' 
estimation.  Unlike  the  military  figure  above,  the  estimated  utilizations  between  the 
supervisor  and  employee  largely  trend  in  a  similar  fashion.  Only  in  one  or  two 
categories  does  the  time  vary  to  a  significant  amount.  The  following  paragraphs 
discuss  the  specific  difference  between  various  employment  categories. 
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Figure  6:  Time  Allocation  Comparison  for  Civilian  Against  Supervisor  Expectation 

While  ignoring  the  supervisors'  indications  of  contractor  time  utilization, 
comparing  supervisor  impression  against  employee  average  utilization  in  many 
categories  is  very  similar.  Utilizing  earned  value  [EV]  data  in  support  of  creating 
estimates  is  essentially  the  same  for  all.  However,  there  are  some  large  disparities 
in  the  results.  Table  9  below  shows  the  difference  between  the  supervisor 
impression  of  time  utilization  and  the  employee  indication  of  activities.  The  table  is 
highlighted  in  pink  for  any  areas  where  the  supervisor  and  employee  disagree  by 
5%  or  greater.  The  category  "analyzing  EVM  data"  indicates  that  the  military  feel  on 
average  that  they  spend  9%  less  time  than  what  the  supervisor  estimates. 
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Table  9:  Time  Comparison  of  Supervisor  Expectation  Versus  as  Reported  by  Employee 


Difference  of  Opinion 

Civilian 

Military 

Analyzing  EVM  Data 

5% 

9% 

Using  EV  data 

1% 

1% 

Non-cost  related  FM 

2% 

7% 

Managing  support  KTRs 

3% 

0% 

Consulting  with  PM 

1% 

5% 

Initial  estimates 

5% 

2% 

Reconciling  estimates 

3% 

7% 

Reviewing  estimates 

3% 

6% 

Post-estimate  docs 

1% 

3% 

What-if  drills 

2% 

3% 

Meetings 

1% 

5% 

Cost  related  training 

4% 

6% 

Non-cost  training 

0% 

6% 

Other 

4% 

13% 

The  table  above  suggests  that  supervisors  maintain  a  strong  understanding 
of  the  way  civilian  employees  spend  their  time.  However,  in  the  case  of  military 
employees,  supervisors’  expectations  seem  to  differ  significantly  from  the 
employees’  reported  usage.  Table  9  above  shows  that  supervisors  over  or 
underestimate  military  time  by  5%  in  9  of  the  14  categories.  It  is  outside  the 
capability  of  this  study  to  determine  the  exact  misunderstanding  for  each  category 
since  the  supervisors  summarize  utilization  per  personnel  and  not  on  individual 
employees.  However,  the  questionnaire  does  present  useful  information,  which 
allows  for  further  exploration  of  time  allocation.  In  the  case  of  one  military 
respondent,  the  only  category  to  receive  a  percentage  weight  was  "Other”  as  the 
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individual  deployed  to  an  overseas  location  and  did  not  complete  any  cost 
estimating  activities. 

The  employee  versus  supervisor  disparity  in  time  expectations  continues  in 
many  areas  when  analyzing  the  data  on  a  base  level.  Figure  7  below  shows  the 
results  of  the  civilian  employees  from  Los  Angeles  Air  Force  Base  [LAAFB)  versus 
the  supervisors’  expectations.  The  data  includes  10  civilian  employees  and 
supervisors’  rating  on  7  civilian  subordinates.  Interestingly  the  supervisors  indicate 
civilian  employees  spend  a  little  over  30%  of  time  analyzing  EVM  data.  The  civilian 
employees  report  only  17%  of  time  spent  analyzing  EVM  data.  The  other  interesting 
disparity  resides  in  initial  estimates.  Again,  supervisors  expect  a  large  percentage  of 
time  spent  in  this  category.  The  difference  of  almost  8%  between  the  supervisors 
and  employees  causes  concern. 
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LAAFB:  Supervisors  Expectations  - 

Civilian 
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Figure  7:  Time  Usage  Expectations  at  LAAFB 

In  every  location  analyzed,  at  least  one  category  varies  by  a  substantial 
percentage  between  the  supervisor  and  employee  report.  Initial  indications  point  to 
a  lack  of  awareness  between  the  supervisor  and  the  employee.  However,  a 
matched-pair  analysis  is  not  possible  in  the  limited  data  set.  The  supervisors  at 
LAAFB  might  be  reporting  time  usage  of  employees  different  from  the  employees 
who  completed  the  survey.  Appendix  E  contains  the  data  in  percentage  of  time 
broken  out  by  base. 

Experience 

A  second  area  highlighted  by  the  RAND  study  is  experience  or  lack  of 
experience  within  the  cost  community.  The  RAND  census  gathered  slightly  different 
information  in  both  personnel  and  in  experience  levels.  However,  comparing  the 
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two  studies  highlights  some  interesting  differences.  While  the  studies  are  purely 
snapshots  in  time,  the  fact  that  the  Cost  Community  Questionnaire  followed  the 
RAND  census  by  almost  three  years  allows  for  a  longitudinal  look  at  any  changes 
that  may  have  taken  place  in  the  acquisition  community.  Figure  8  below  shows  the 
RAND  census  information  in  numbers  of  personnel.  RAND  summarized  its 
information  regarding  personnel  as  either  organic  or  contractor.  The  organic  group 
contained  both  military  and  government  civilians.  The  RAND  census  maintained 
similar  numbers  of  both  organic  personnel  and  contractors,  with  the  total  number 
being  358  people  split  evenly  [180  organic  and  178  contractors). 


Figure  8:  Number  of  Personnel  per  Experience  Bin  as  Reported  in  RAND  Census  (Vernez  &  Massey,  2009) 


Figure  8  above  also  shows  a  large  disparity  between  the  number  of  organic 
personnel  with  five  years  or  less  of  experience  and  that  of  the  contractors.  Figure  9 
below  reformats  the  information  into  percentage  of  personnel  based  on  the  number 
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of  years  of  experience.  Contractors  maintain  around  25%  of  personnel  in  each  of 


the  four  experience  groupings.  However,  as  figure  9  below  highlights,  organic 


personnel  are  nowhere  near  as  evenly  distributed.  Individuals  with  five  years  or 


less  of  experience  account  for  over  50%  of  the  total  organic  workforce. 


RAND  Census 

60% 

g  50% 

o 

I  40% 

a 

o  30% 

o 
bX) 

S  20% 

c 
o 

“  10% 
o 

On 

0% 


Figure  9:  Percentage  of  Personnel  per  Experience  Bin  as  Reported  in  RAND  Census  (Vernez  &  Massey, 

2009) 


The  reason  for  discussing  the  RAND  census  data  is  to  establish  a  point  of 
comparison.  Figure  10  below  shows  the  information  gathered  from  the  2010 
survey.  Civilians  comprise  the  bulk  of  the  cost  community  with  a  ratio  of  5  to  1 
versus  the  military  who  completed  the  questionnaire.  Figure  10  includes 
supervisors  with  the  idea  that  a  more  experienced  supervisor  will  allow  for  a  more 
complete  cost  analysis  team.  The  number  of  civilians  in  each  category  of  experience 
seems  to  mimic  a  learning  curve  function  utilized  in  cost  estimation. 
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Figure  10:  Number  of  Personnel  per  Experience  Bin  as  Indicated  in  Questionnaire 

The  Pareto  chart  [Figure  11)  below  summarizes  the  experience  contained  in 
the  cost  analysis  community  as  percentages.  The  percentage  of  military  cost 
analysts  with  five  years  or  less  of  cost  experience  is  an  alarming  81%.  The  civilians 
fare  much  better  at  a  41%  rate  in  the  same  category.  As  represented,  no  military 
with  greater  than  10  years  of  experience  completed  the  questionnaire.  The  line 
increasing  towards  the  right  represents  the  cumulative  total  of  military  and  civilian 
analysts  in  non-supervisory  positions.  Non-supervisory  personnel  with  10  years  or 
less  of  experience  account  for  70%  of  the  total  cost  estimating  organic  workforce. 
Only  17%  of  the  workforce  had  greater  than  15  years  of  experience.  Even  when 
including  the  supervisory  personnel  in  the  total,  approximately  64%  of  personnel 
have  10  years  or  less  of  experience.  Supervisors  raise  the  percentage  of  the 
workforce  with  greater  than  15  years  of  experience  from  17%  to  21%. 
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Figure  11:  Percentage  of  Personnel  per  Experience  Bin  as  Indicated  in  Questionnaire 


Comparing  the  point  estimate  created  in  the  RAND  census  and  the  recently 
acquired  data  creates  a  longitudinal  view.  Figure  12  below  shows  the  RAND  census 
in  red  and  the  cost  questionnaire  results  in  blue  as  percentage  of  the  total  organic 
workforce.  For  comparison  against  the  RAND  study,  which  included  "cost  leads"  in 
the  organic  personnel,  1  include  the  supervisory  personnel  in  the  questionnaire 
numbers.  Within  the  category  of  five  years  of  experience  or  less,  the  RAND  census 
reported  a  higher  percentage  than  the  questionnaire.  As  such,  the  questionnaire 
reports  a  higher  percentage  of  personnel  in  both  the  6-10  and  11-20  year  range. 

One  explanation  for  this  is  that  some  of  the  individuals  remaining  in  the  career  field 
and  gaining  years  of  experience  between  the  two  studies.  The  personnel  with 
greater  than  20  years  of  experience  remains  largely  unchanged  between  the  two 
studies. 
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Figure  12:  Comparison  of  RAND  Census  Versus  Questionnaire 


Training 

For  the  purposes  of  the  online  questionnaire.  Acquisition  Professional 
Development  Program  (APDP)  levels  serve  as  a  proxy  for  training.  Of  all 
supervisors,  85%  hold  APDP  in  at  least  one  category  while  only  76%  of  non- 
supervisory  personnel  hold  at  least  one  level  of  APDP  of  any  certificate.  Figure  13 
below  shows  that  approximately  70%  of  supervisors  maintain  an  APDP  certification 
of  some  level  in  the  cost  specialty.  The  column  indicating  both  career  field  cost 
estimating  (BCF-CE]  and  financial  management  [BCF-FM]  APDP  levels  shows  the 
lowest  level  of  either  category.  If  a  supervisor  indicated  a  BCF-CE  level  2  and  a  BCF- 
FM  level  3,  the  figure  below  categorizes  this  as  a  level  2  in  both.  Over  30%  of 
supervisors  indicated  both  a  BCF-CE  and  a  BCF-FM. 
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Figure  13:  Supervisor  APDP  Training  Levels  as  Reported  in  Questionnaire 


Non-supervisory  personnel  reported  that  approximately  60%  maintain  at 
least  some  level  of  BCF-CE  level.  As  with  Figure  13  above,  Figure  14  summarizes  the 
results  for  APDP  training  levels.  Compared  to  supervisors,  a  larger  percentage  of 
non-supervisory  personnel  maintain  both  BCF-CE  and  BCF-FM.  Of  non-supervisory 
personnel,  35%  reported  they  completed  some  level  of  certification  in  both  BCF-CE 
and  BCF-FM.  The  lowest  level  of  certificate  indicated  by  the  respondent  comprises 
the  levels  show  in  the  "both"  column. 
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Figure  14:  Non-Supervisor  APDP  Training  Levels  as  Reported  in  Questionnaire 


Additionally,  over  50%  of  supervisors  either  reported  having  a  certified 
defense  financial  management  [CDFM]  or  certified  cost  estimator/analyst  [CCEA] 
certificate.  Only  20%  of  non-supervisory  personnel  indicated  a  certification,  split 


evenly  between  CDFM  and  CCEA. 
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Figure  15:  Experience  Level  Compared  to  Maximum  APDP  Certificate  Level  -  CE 
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Figure  15  above  expands  upon  the  analysis  of  training  within  the  cost 
community.  The  y-axis  represents  the  percentage  of  certificates  per  level  with  the  x- 
axis  indicating  the  years  of  experience  indicated  by  respondents.  The  CE-level  1 
certificate  decreases  at  an  almost  linear  rate  with  by  workers  with  5  years  or  less  of 
experience  holding  almost  50%  of  all  CE-level  1  certificates.  Again,  the  CE-level 
indicted  in  the  figure  is  the  maximum  held,  meaning  that  if  the  analyst  indicated  a 
level  2  certificate  then  it  is  assumed  the  worker  maintains  a  level  1,  but  the  level  1 
certificate  is  not  indicated  in  the  figure  above.  The  percentage  of  level  3  certificates 
does  not  follow  the  expected  path.  However,  with  only  six  total  certificates;  one 
certificate  represents  almost  17%  of  the  total. 

Supervisor  Feedback 

As  previously  stated,  approximately  20  supervisors  completed  the 
questionnaire.  These  20  supervisors  indicated  that  they  supervise  a  total  of  109 
civilians,  13  military,  and  59  contractors.  The  questionnaire  for  the  supervisors 
includes  some  qualitative  questions  seeking  to  identify  any  universal  problems  or 
opportunities.  All  totaled,  the  supervisors  indicated  they  assisted  in  224  different 
programs  broken  out  as  42  acquisition  category  1  (ACAT  1],  27  ACAT  11  and  171  non- 
ACAT  1/11  programs. 

Over  73%  of  the  supervisors  indicated  that  they  did  not  have  sufficient 
numbers  of  analysts  to  complete  the  required  workload.  Additionally,  while  not 
explicitly  asked,  three  supervisors  indicated  their  office  works  considerable 
overtime  in  order  to  meet  program  goals.  Of  those  supervisors  indicating  they  did 
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have  sufficient  work  force,  four  out  of  the  five  indicated  their  workforce  lacked 
experience.  All  totaled,  82%  of  the  supervisors  mentioned  at  least  once  that 
experience  in  cost  analysis  was  lacking.  In  addition  to  the  lack  of  experience,  63% 
indicated  their  team  did  not  have  a  sufficient  level  of  technical  competence.  Of  the 
respondents,  75%  indicate  sacrificing  tasks  in  order  to  maintain  current  production 
levels. 

When  questioned  specifically  on  aspects  of  WSARA,  supervisors  answered  in 
a  largely  positive  way  with  79%  feeling  that  WSARA  addresses  some  of  the 
deficiencies  in  the  cost  community.  The  general  feeling  was  that  WSARA  placed 
increased  pressure  on  decision  makers  to  utilize  the  cost  estimates  created  by  the 
organizations.  A  few  supervisors  did  indicate  that  while  largely  positive  in  its 
effects,  WSARA  was  detrimental  to  the  organization.  One  supervisor  who  indicated 
that  the  office  needed  more  individuals  prior  to  the  WSARA  implementation  stated 
that  the  new  requirements  created  an  even  greater  need  for  increased  work  force. 
Compounding  the  effect  of  raw  manning  number,  hiring  difficulties  and  personnel 
turnover  create  a  need  for  training  and  result  in  knowledge  drain.  While  hiring  over 
69  individuals  during  the  preceding  24  months  of  the  study,  the  majority  of  the 
respondents  (94%)  indicated  hiring  new  analysts  as  difficult,  with  50%  indicating 
hiring  as  extremely  difficult.  Many  respondents  specified  recently  filling  a  few 
personnel  billets,  however,  many  remained  unfilled. 


55 


Preliminary  analysis 

The  purpose  of  preliminary  analysis  is  to  perform  a  confirmatory  analysis  of 
the  individual  measures  and  construct  the  four-factor  measure  of  virtualness.  As 
identified  earlier,  current  literature  lacks  a  commonly  accepted  measure  for 
virtualness  and  as  such,  this  paper  seeks  to  partially  address  this  point.  The  findings 
will  highlight  that  significant  correlations  between  various  items  indicate  that  the 
questionnaire  exhibits  required  power. 

Initial  virtualness  measure  creation 

The  questionnaire  contains  41  individual  questions  addressing  the  four 
dimensions  of  virtualness.  Preliminary  analysis  of  these  items  indicates  that  a  40- 
question  measure  failed  to  achieve  specificity.  In  order  to  identify  the  strongest 
measures,  1  first  performed  a  factor  analysis  within  each  dimension  of  virtualness. 
Seeking  to  create  a  16-question  measure  for  virtualness,  1  identified  the  four 
strongest  measures  within  each  dimension.  Table  10  below  summarizes  the 
strongest  four  items  in  each  dimension  of  virtualness.  After  identifying  the  four 
questions  in  each  dimension,  1  conducted  reliability  analyses  on  the  items. 
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Table  10:  Confirmatory  Factor  Analysis  -  Initial  16item  Measure  for  Virtualness 


Virtualness  Factor  Analysis  - 16  item  measure  (preliminary) 

Technical  Support 

Synchronicity 

Initial  Eigenvalue 
Technical  Support  4  0.848 

Technical  Support  3  0.793 

Technical  Support  10  0.668 

Technical  Support  7  0.470 

Initial  Eigenvalue 
Synchronicity  9  0.846 

Synchronicity  2  0.834 

Synchronicity  8  0.820 

Synchronicity  6  0.768 

Richness  of  Technology 

Distance 

Initial  Eigenvalue 
Tech  Richness  7  0.788 

Tech  Richness  10  0.749 

Tech  Richness  5  0.734 

Tech  Richness  1  0.687 

Initial  Eigenvalue 
Distance  9  0.902 

Distance  8  0.890 

Distance  1  0.872 

Distance  6  0.660 

*  Extraction  Method:  Principal  Component  Analysis 


Table  11  below  summarizes  the  reliability  statistics.  The  commonly  accepted 
threshold  for  reliability  is  greater  than  0.70  Cronbach’s  alpha.  As  the  table  indicates, 
the  preliminary  analysis  for  three  of  the  four  measures  exceeds  the  desired  0.70. 
However,  the  reliability  of  reliance  on  technical  support  is  low.  Item  7  of  technical 
support  exhibits  a  mediocre  relation  to  the  other  questions,  resulting  in  a  low 
Eigenvalue.  Technical  support  item  7  may  be  starting  to  measure  a  secondary  factor 
of  technical  support  that  the  other  three  questions  do  not.  Additionally,  while  the 
sample  size  is  statistically  significant,  the  weak  Cronbach's  alpha  may  be  in  part  due 
to  the  relatively  small  sample  size. 
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Table  11:  Reliability  statistics  -  Initial  16  item  measure  for  virtualness 


Virtualness  reliability  statistics  - 

16  item  measure  (preliminary) 

Cronbach's  Alpha 

Distance 

0.853 

Technical  Support 

0.670 

Richness  of  Technology 

0.709 

Synchronicity 

0.834 

structural  Equation  Modeling 

Having  identified  the  four  items  within  the  four  dimensions  that  I  would 
utilize  to  construct  the  measure  for  virtualness,  1  proceeded  to  complete  a  structural 
equation  model  (SEM)  to  verify  the  applicability  of  the  measure.  Structural  equation 
modeling  consists  of  three  primary  processes:  path  analysis,  confirmatory  factor 
analysis,  and  structural  regression  models  all  comprise  aspects  of  the  SEM.  SEM  is 
excellent  at  validating  that  data  fit  a  given  model;  however,  SEM  is  not  an 
exploratory  technique  used  to  create  a  model.  Given  that  1  had  arrived  at  a 
preliminary  model,  which  used  four  questions  in  each  of  the  four  dimensions  of 
virtualness,  1  leveraged  SEM  to  verify  the  applicability  of  the  model. 

When  implementing  SEM  to  analyze  the  16  measures  for  virtualness,  1  had  to 
acknowledge  that  while  each  dimension  is  separate,  some  overlap  might  exist. 

While  the  face  validity  of  a  question  on  distance  may  indicate  a  single  factor,  the 
dimension  of  distance  may  affect  or  be  affected  by  the  level  of  synchronicity.  After 
addressing  the  possibility  for  interdependence  of  the  measures,  1  ran  a  model  fit 
analysis.  The  preliminary  16-question  measure  for  virtualness  proved  acceptable. 
The  relative  chi-square,  which  is  chi-square  divided  by  the  degrees  of  freedom, 
resulted  in  1.606.  The  incremental  fit  index  (IFl)  of  0.917  for  the  16-item  measure 
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exceeded  the  common  threshold  of  0.9.  The  comparative  fit  index  (CFI)  of  0.913 
surpassed  the  desired  0.9  figure.  However,  the  root  mean  square  error  of 
approximation  (RMSEA]  for  the  measure  resulted  in  a  marginal  fit  of  0.076  while  a 
desired  fit  is  near  0.05. 

Final  Analysis  -  Virtualness  measure 

Preliminary  analysis  of  the  SEM  output  indicated  that  the  measure  needed 
improvement.  The  acceptable  relative  chi-square  of  1.606  and  the  marginal  RMSEA 
of  0.076  highlighted  the  need  for  more  refinement  of  the  measure  for  virtualness.  In 
order  to  improve  model  fit,  1  reduced  the  number  of  items  per  factor  from  four  to 
three.  Rerunning  the  proper  confirmatory  factor  analysis  and  reliability  tests 
allowed  for  an  overall  reduction  in  items  from  16  to  13.  Table  12  below  summarizes 
the  confirmatory  factor  analysis  for  the  final  13 -item  measure  of  virtualness.  The 
initial  Eigenvalues  for  the  individual  items  generally  improved  when  moving  from 
the  16-item  to  the  13-item  measure. 


59 


Table  12:  Confirmatory  Factor  Analysis  -  Final  13  item  Measure  for  Virtualness 


Virtualness  factor  analysis  -  13-item  measure  (final) 

Technical  Support 

Synchronicity 

Initial  Eigenvalue 
Technical  Support  4  0.879 

Technical  Support  3  0.793 

Technical  Support  10  0.698 

Initial  Eigenvalue 
Synchronicity  9  0.855 

Synchronicity  2  0.850 

Synchronicity  8  0.862 

Richness  of  Technology 

Distance 

Initial  Eigenvalue 
Tech  Richness  7  0.788 

Tech  Richness  10  0.749 

Tech  Richness  5  0.734 

Tech  Richness  1  0.687 

Initial  Eigenvalue 
Distance  9  0.932 

Distance  8  0.855 

Distance  1  0.904 

*  Extraction  Method:  Principal  Component  Ana 

ysis 

The  goal  of  reducing  the  items  from  16  to  12  was  not  achievable  due  to 
reliability  concern.  All  dimensions  except  for  richness  of  technology  exceeded  the 
0.7  threshold  for  Cronbach’s  Alpha  after  reducing  the  number  of  items  per  measure 
from  four  to  three.  Richness  of  technology  required  four  items  in  order  to  keep  the 
Cronbach’s  Alpha  above  0.7.  Table  13  below  summarizes  the  reliability  statistics  for 
the  final  13-item  measure  of  virtualness. 


Table  13:  Reliability  Statistics  -  Final  13  item  Measure  for  Virtualness 


Virtualness  reliability  statistics 

- 13  item  measure  (final) 

Cronbach's  Alpha 

Distance 

0.889 

Technical  Support 

0.705 

Richness  of  Technology 

0.709 

Synchronicity 

0.816 

While  the  13-item  measure  meets  all  desired  statistics,  having  an  uneven 
number  of  items  for  the  individual  dimensions  creates  an  aggregation  problem. 
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Aggregating  items  individually  into  a  single  "virtualness”  item  creates  a 
disproportional  weighting  with  Value  weighing  more  heavily  in  the  resulting  level  of 
virtualness.  In  order  to  avoid  this  problem  of  weighting,  1  first  aggregated  the  items 
into  the  respective  dimension.  1  then  aggregated  the  resulting  four  dimensions  of 
virtualness  into  a  single  measure  for  virtualness.  The  aggregated  measure  follows 
the  Likert-type  scale  with  1  being  low  virtualness  and  5  being  highly  virtual.  A 
result  of  "highly  virtual”  is  similar  to  a  very  disruptive  environment.  For  analysis 
purposes,  1  reverse-coded  the  virtual  measure  due  to  the  idea  that  a  highly  virtual 
measure  should  negatively  correlate  to  the  other  measures  in  the  questionnaire. 

Table  14:  Final  13  item  Measure  for  Virtualness 

_ Distance _ 

1.  My  supervisor  thoroughly  understands  my  working  environment. 

2.  My  supervisor  and  I  have  a  common  understanding  of  work  requirements. 

3.  My  supervisor  understands  the  daily  requirements  of  my  job. _ 

_ Technical  Support _ 

1.  My  supervisor  carries  a  cell  phone. 

2.  My  supervisor  is  available  on  a  cell  phone  throughout  the  workday. 

3.  I  utilize  information  technology  in  my  daily  interactions  with  my  supervisor. 

_ Richness  of  Technology _ 

1.  I  am  easily  able  to  understand  the  message  from  my  supervisor. 

2.  I  am  easily  able  to  understand  a  variety  of  different  cues  (e.g.  emotional  tone, 
feelings)  from  my  supervisor. 

3.  I  am  easily  able  to  tailor  my  messages  to  my  supervisor. 

I  am  able  to  use  rich  and  varied  language  when  communicating  with  my 

4.  supervisor. _ 

Synchronicity 

1.  It  is  often  difficult  to  get  in  touch  with  my  supervisor. 

2.  My  supervisor  and  I  have  difficulties  aligning  our  schedules. 

3.  My  supervisor  is  available  whenever  I  need  him/her. 
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After  identifying  the  13-item  measure  and  aggregating  the  data,  I  utilized 
SEM  a  second  time  to  test  model  fit.  The  new  13-item  measure  proved  much  better 
in  all  tests.  The  relative  chi-square  improved  from  1.606  to  1.303.  The  IFl  and  CFl 
improved  as  well  from  0.917  to  0.969  and  from  0.913  to  0.967  respectively.  The 
final  hurdle  of  RMSEA  proved  to  solidify  the  improved  measure,  falling  from  0.076 
to  0.053.  Figure  16  below  shows  the  final  13-item  measure  as  tested  in  SEM  for 
model  fit. 


Figure  16: 13-item  Measure  for  Virtualness  (final)  Amos  SEM 


62 


Hypotheses  testing 

As  previously  mentioned,  over  300  electronic  invitations  were  sent  with  an 
approximate  response  rate  of  34%.  All  measures  contained  over  100  responses. 
Table  15  below  shows  the  descriptive  information  for  the  calculated  values.  As 
example,  the  measure  for  job  satisfaction  uses  four  questions.  Averaging  the 
responses  for  the  four  questions  into  an  overall  Likert-type  scale  creates  a  value  per 
individual. 


Table  15:  Descriptive  Statistics 


Descriptive  Statistics 

N 

Min 

Max 

Mean 

Std.  Dev 

Jobsat 

102 

1.00 

5.00 

3.96 

0.82 

LMX 

103 

1.00 

5.00 

3.87 

0.86 

Org  Commit 

102 

1.00 

4.75 

3.29 

0.72 

Trust 

105 

1.29 

4.71 

3.50 

0.64 

Turnover  Intention 

102 

1.00 

5.00 

2.11 

0.95 

Virtualness 

102 

1.06 

3.31 

2.12 

0.50 

Table  16  below  summarizes  the  correlations  between  the  measures.  All 
measures  indicate  a  strong  correlation  with  virtualness.  The  matrix  highlights  the 
relationship  between  the  variables  measured  in  the  questionnaire.  As  expected, 
turnover  intention  negatively  correlates  with  all  measures  except  virtualness. 
Virtualness  exhibits  similar  correlations  with  the  other  measures  in  that  it 
negatively  correlates.  The  negative  correlation  indicates  that  as  virtualness 
increases,  job  satisfaction,  LMX,  and  organizational  commitment  decrease. 
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Table  16:  Pearson  Correlation  Matrix 


Correlations 

Trust 

Job  sat 

LMX 

Turn  Int 

Org 

Commit 

Virtualness 

Trust 

(.73) 

Job  Satisfaction 

** 

.55 

(.85) 

Leader-Member 

Exchange 

** 

.78 

.56** 

(.95) 

Turnover  Intention 

** 

-.50 

-.82** 

-.47** 

(.64) 

Organizational 

Commitment 

** 

.57 

.71** 

.51** 

-.76** 

(.87) 

Virtualness 

** 

-.67 

-.57** 

-.78** 

** 

.44 

-.49** 

(.79) 

Reliabilities  appear  in  parentheses  along  the  diagonal 
**.  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


Table  17  below  summarizes  the  results  of  multiple  regression  analyses.  In 
order  to  test  hypotheses  1-4,  the  independent  variable,  virtualness,  was  regressed 
against  the  dependent  variable.  In  each  case,  the  regression  consisted  of  one 
independent  and  one  dependent  variable.  All  hypotheses  proved  significant  and 
exhibited  a  medium  effect  size  on  the  dependent  variable.  The  effect  size  for  the 
relationship  between  virtualness  and  trust  tested  more  significant  than  expected. 
For  hypotheses  5-7,  the  independent  variable  differed  during  each  regression  while 
testing  against  the  singular  dependent  variable  of  turnover  intention.  Again,  all 
indications  tested  as  expected.  The  effect  sizes  for  job  satisfaction  and 
organizational  commitment  in  relation  to  turnover  intention  regressed  more 
strongly  than  expected. 
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Table  17:  Multiple  Regression  Analysis  for  Hypothesis  Testing 


Regression  Analysis  of  Hypotheses 

Variabie 

Standardized 

independen 

Adj. 

t- 

Hypothesis 

t 

Dependent 

3 

SE  3 

R^ 

vaiue 

Sig. 

1 

Virtuainess 

Trust 

-0.67 

0.10 

0.45 

0.45 

-9.11 

0.00 

2 

Job  Satisfaction 

-0.57 

0.13 

0.32 

0.32 

-6.89 

0.00 

3 

Org  Commit 

-0.49 

0.13 

0.24 

0.23 

-5.59 

0.00 

4 

Turnover 

0.44 

0.17 

0.20 

0.19 

4.94 

0.00 

intention 

Variabie 

Standardized 

independen 

Adj. 

t- 

Hypothesis 

t 

Dependent 

3 

SE  p 

R^ 

vaiue 

Sig. 

5 

Trust 

Turnover 

-0.50 

0.13 

0.25 

0.24 

-5.71 

0.00 

6 

Job  Sat 

intention 

-0.82 

0.07 

0.67 

0.67 

-14.27 

0.00 

7 

Org  Commit 

-0.76 

0.09 

0.58 

0.57 

-11.70 

0.00 

Testing  mediation  effects  of  the  variables  required  a  three-step  process 
(Frazier,  Tix,  &  Barron,  2004].  First,  the  independent  variable  of  virtualness  was 
regressed  against  the  dependent  variable  turnover  intention.  Figure  17  below 
diagrams  the  tested  regression  for  Hypothesis  8.  The  results  of  Step  1  are  included 
in  Table  17  above.  The  need  to  complete  Step  1  originated  from  the  need  to  identify 
if  any  relationship  existed  for  which  the  mediator  might  affect.  If  Step  1  returned 
values  of  no  significance  or  no  effect  size,  then  the  need  to  test  mediation  was  moot. 


Virtualness 

Turnover 

Intention 

Figure  17:  Step  1  Mediation  testing 
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Step  2  of  the  mediation  testing  process  involved  regressing  the  independent 
variable  against  the  proposed  mediator.  Step  2  was  critical  to  establish  a 
relationship  between  the  independent  variable  and  the  proposed  mediator.  Figure 
18  below  diagrams  the  process  of  regressing  virtualness  against  trust  for 
Hypothesis  8.  Again,  the  results  of  step  two  are  included  in  Table  17  above. 


Virtualness 


Trust 


Turnover 

Intention 


Figure  18:  Step  2  Mediation  testing 

The  third  step  involved  regressing  the  independent  variables  against  the 
dependent  variable.  For  Hypothesis  8,  the  regression  analysis  included  independent 
variables  virtualness  and  trust  against  the  dependent  variable  turnover  intention. 
Figure  19  below  diagrams  the  relations  tested  in  the  multiple  regression  analysis. 
Comparing  the  regression  analysis  completed  in  Test  3  against  the  previous  tests 
helps  to  identify  any  effects  of  mediation. 


r 


Virtualness 


Turnover 

Intention 


Figure  19:  Step  3  Mediation  testing 

Frazier  et  al.  [2004]  indicate  that  in  order  for  the  mediation  of  the 
independent  variable’s  relation  with  the  dependent  variable,  the  Beta  [P]  of  the 
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independent  variable  must  be  zero  in  the  multiple  regression  analysis.  Stated 
differently,  the  reduction  of  p  from  the  determined  p  of  Step  1  to  zero  in  Step  3 
indicates  a  complete  mediation  of  the  relationship  between  the  independent  and 
dependent  variables.  Table  18  below  summarizes  the  regressions  performed  to 
complete  Step  3.  None  of  the  p  associated  with  virtualness  in  hypotheses  8-10 
reduced  to  zero  when  introducing  a  mediator.  Standardizing  all  p  ensures  proper 
basis  for  comparison.  However,  when  comparing  the  P  for  virtualness  in  Step  3 
against  the  corresponding  Step  1  p,  all  p  decreased.  Given  that  none  of  the  p 
reduced  to  zero,  the  testing  for  mediation  requires  additional  inspection. 


Table  18:  Multiple  Regression  analysis  for  mediation  testing 


Results  of  multiple  regression  analysis 

Variable 

Standardized 

Test  for 

Hypothesis 

Independent 

Dependent 

p 

SEP 

R" 

Adj. 

t-value 

Sig. 

Mediation 

8 

Trust  * 

Turnover 

-0.36 

0.17 

0.27 

0.25 

-3.10 

0.00 

2.00 

Virtualness 

Intention 

0.20 

0.22 

1.72 

0.09 

9 

Job  Sat  * 

Turnover 

-0.84 

0.08 

0.67 

0.67 

-11.96 

0.00 

3.92 

Virtualness 

Intention 

-0.03 

0.13 

-0.45 

0.65 

10 

Org  Commit  * 

Turnover 

-0.71 

0.10 

0.59 

0.58 

-9.62 

0.00 

3.42 

Virtualness 

Intention 

0.10 

0.14 

1.28 

0.20 

The  procedure  for  determining  mediation,  given  that  p  was  not  zero, 
involved  using  the  information  derived  in  Steps  2  and  3.  To  complete  the  equation 
below,  the  variable  "a"  corresponds  to  the  p  of  Step  2. 


Equation  1:  Testing  for  mediation  (Kenny,  Kashy,  &  Bolger,  1998) 
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The  b  in  the  equation  equals  the  P  found  in  Step  3  relating  to  the  proposed 
mediator  and  the  dependent  variable.  The  equation  below  includes  the  standard 
error  for  each  corresponding  relationship.  The  resulting  z  value  allowed  for 
statistical  significance  testing,  in  this  case  a  95%  confidence  interval.  Table  18 
above  includes  the  results  of  the  mediation  test  for  each  of  the  hypotheses.  In  all 
cases,  the  values  exceeded  the  required  value  of  1.96,  indicating  the  mediation  was 
significant.  The  results  of  the  test  indicated  that  the  data  supports  hypotheses  8-10. 


68 


V.  Conclusion  and  Recommendations 


The  primary  goal  of  the  questionnaire  was  threefold.  First,  the  questionnaire 
sought  to  identify  the  areas  of  cost  estimating  currently  requiring  the  greatest 
amount  of  time  within  the  community.  Second,  1  sought  to  identify  any  differences 
from  the  previous  snapshot  in  time  summarized  in  the  RAND  study.  Third,  1  sought 
to  develop  a  reliable  measure  for  virtualness  and  to  use  this  measure  with  hopes  of 
highlighting  areas  for  improvement  or  strengths  currently  exhibited  in  the  cost 
community. 

The  questionnaire  succeeded  in  identifying  the  average  time  use  for  cost 
estimators.  As  shown  in  Table  4  above,  the  four  largest  areas  of  time  utilization  for 
cost  estimators  are  consulting  with  the  PM,  preparing  initial  estimates,  reviewing 
estimates,  and  meetings.  A  secondary  result  of  the  questionnaire  raises  concern 
about  the  actual  understanding  of  time  use  between  supervisors  and  subordinates. 
The  large  variation  between  expectations  of  the  supervisor  and  stated  workload  of 
the  subordinate  seems  to  show  a  substantial  disconnect.  The  largest  disparity  exists 
between  supervisors  and  military  subordinates.  Some  of  the  comments  of  the 
supervisors  may  explain  this  disparity,  as  one  supervisor  wrote,  "The  military  are 
never  here  more  than  2  years  total  time  in  the  branch,  and  during  that  2 -year 
period,  they  all  must  deploy  for  180  days.  Add  on  the  two  months  of  pre¬ 
deployment  training  and  the  two  months  of  post-deployment  re-orientation  and 
leave,  and  they  are  pretty  much  useless  to  the. ..mission."  If  representative,  the 
statement  indicates  the  frustrations  felt  by  the  supervisors  towards  the  military 
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operations  tempo  and,  as  a  result,  supervisors  take  a  more  hands-off  approach  to 
the  management  of  military  personnel. 

A  second  area  of  concern  surrounding  the  time  usage  as  reported  by  the 
respondents  is  in  the  area  of  EVM.  Using  EVM  data  and  analyzing  EV  data  accounts 
for  less  than  10%  total  time  within  each  personnel  category.  Given  the  current 
focus  on  accurate  cost  estimates  and  management  controls,  the  respondents  spend 
very  little  time  reviewing  program  performance.  WSARA  stresses  the  importance  of 
accurate  cost  estimates  with  the  goal  of  more  stringent  program  control.  In  reality, 
no  matter  how  accurate  the  cost  estimate,  improper  control  and  management  of  the 
program  strike  even  the  best  cost  estimates  useless. 

The  second  area  of  focus  for  the  questionnaire  is  its  comparison  against  the 
RAND  data.  When  comparing  the  personnel  numbers  surrounding  experience,  the 
recent  results  seem  positive.  The  percentage  of  personnel  with  less  than  five  years 
of  experience  has  decreased  compared  to  other  categories.  Additionally,  the  area  of 
5  to  20  years  of  experience  has  grown  in  percentage.  While  these  figures  bode  well 
for  the  community,  supervisors  report  that  the  growth  is  not  sufficient  when 
compared  to  demand:  "Absolutely  we  do  not  have  enough  analysts,"  "We  are  short 
about  18  people.”  "1  need  at  least  one  additional  analyst  per  program,"  are  just  a  few 
of  the  comments  from  supervisors  when  asked  whether  they  had  enough  cost 
analysts  to  meet  the  program  demand. 

The  situation  grows  drearier  when  respondents  report  on  experience:  "We 
have  much  to  learn;  three  fourths  of  our  staff  are  neophytes,”  "Our  staff  has  either 
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over  10  years. ..or  less  than  2,"  "too  inexperienced,”  "75%  of  my  staff  has  three  years' 
experience  or  less;  50%  of  my  staff  is  within  their  first  year."  These  statements  are 
just  a  snippet  of  the  unrest  within  the  community  regarding  experience  and 
workforce.  While  WSARA  and  other  recent  policies  stress  the  importance  of 
revitalizing  the  acquisition  community,  many  areas  experience  difficulty  filling 
empty  billets.  While  1  previously  discussed  the  recent  hiring  in  personnel,  the 
supervisors  reported  37  people  leaving  the  cost  community  during  the  same  period. 
The  net  result  of  32  new  individuals  seems  to  correspond  with  some  of  the  above 
statements  about  inexperienced  staff 

Additionally,  while  the  5-20  years  of  experience  group  grew  as  percentage, 
the  over-20  group  remained  largely  unchanged.  This  could  indicate  that  retention 
in  the  mid-grade  pay  ranges  is  good,  while  the  upper  band  is  unchanged.  A 
statement  1  heard  multiple  times  was  that  it  takes  at  least  six  years  of  cost 
estimating  to  become  proficient.  As  such,  the  cost  analysis  field  considers  personnel 
with  less  than  10  years  as  junior  or  inexperienced.  The  report  shows  that  70%  of 
the  cost  estimating  community  has  10  years  or  less  of  experience.  Seasoned  or 
senior  analysts  comprise  only  30%  of  the  field  when  using  number  of  years  as  a 
proxy  for  experience  and  ability. 

Third,  the  questionnaire  sought  to  develop  a  measure  for  virtualness.  The 
literature  surrounding  virtualness  differs  upon  whether  three  or  four  dimensions  of 
virtualness  are  required  for  an  accurate  measure.  This  study  argues  that  physical 
distance  is  too  limiting  in  its  relation  to  virtualness  and,  as  such,  a  more  accurate 
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determinate  is  psychological  distance.  Virtualness  can  be  likened  to  discontinuities, 
and  so  the  more  virtual  an  organization,  team,  or  relationship,  the  more 
discontinuous  the  relations.  Psychological  distance  amplifies  the  discontinuity 
between  individuals  and  is  tangentially  necessary  to  virtualness.  The  41-item 
measure  significantly  reduced  to  a  valid,  reliable  13-item  questionnaire. 

Limitations 

While  the  available  sample  set  allowed  for  sufficient  data  to  create  a 
statistically  significant  report,  a  larger  population  would  allow  for  a  more  thorough 
analysis.  In  addition  to  the  smaller  population  of  the  cost  estimating  community 
within  the  Air  Force,  a  non-military  or  government-related  population  would  ensure 
applicability  in  civilian  businesses.  While  the  measure  for  virtualness  satisfied  all 
confirmatory  requirements  for  reliability  and  model  fit,  a  second  sample  to  verify 
the  measure  was  not  applicable  due  to  time  constraints.  As  such,  an  independent 
verification  of  the  measure  would  boost  the  acceptance  of  the  measure. 

The  comparison  between  the  RAND  data  and  my  questionnaire  also  warrants 
comments  on  limitations.  The  data  gathered  in  both  instances  are  samples  of  the 
total  population.  As  such,  there  are  to  be  expected  variances  between  the  samples. 
Natural  sample  variance  might  explain  the  increase  in  experience  observed  in  the 
2010  questionnaire,  resulting  in  essentially  the  same  levels  of  experience  between 
the  two  samples.  However,  dismissing  the  observed  increase  in  middle  level 
experience  merely  due  to  variance  ignores  the  plausibility  of  typical  maturation. 
Over  a  two-year  time  lapse  occurred  between  the  2010  questionnaire  and  the  RAND 


72 


study.  Normal  experience  growth  should  occur  during  the  two  years  subsequent  the 
RAND  study.  Additionally,  the  external  factors  such  as  the  weakened  economy 
might  push  individuals  to  remain  in  government  service  work  longer  than  typical. 
The  economy  is  just  but  one  factor  that  might  contribute  to  a  greater  retention  rate 
that  would  cause  an  increase  in  experience.  Decision  makers  must  apply  the  proper 
retention  tools  to  ensure  adequate  growth  and  maintain  the  current  experience 
observed  throughout  the  cost  estimating  community. 

Further  Research 

As  mentioned  in  the  limitations  section,  an  independent  verification  of  the 
model  would  ensure  reliability.  Subsequent  studies  potentially  improve  the 
measure  through  refinement  and  validation.  Additionally,  utilizing  the  model  to 
verify  antecedents  and  outcomes  previously  studied  in  literature  would  strongly 
support  the  validity  of  thelSitem  measure  for  virtualness. 

While  sampling  as  a  data  gathering  method  is  a  limitation  of  this  research, 
further  samplings  might  verify  the  findings  within  the  cost  community.  The  RAND 
study  and  this  research  create  two  snapshots  in  time  of  the  cost  community.  A 
greater  number  of  studies  would  allow  for  a  greater  understanding  of  the 
community  and  its  allocation  of  resources. 

A  second  possibility  for  further  research  is  developing  the  proper  allocation 
of  time  for  the  cost  estimating  community.  The  questionnaire  gathered  information 
as  to  the  current  allocation  of  time,  but  did  not  develop  a  recommendation  for 
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proper  time  allocation.  One  might  use  the  Delphi  technique  to  define  a  generic 
template  for  cost  analysts  to  model  as  the  recommended  allocation  of  time. 

Lastly,  using  the  measure  of  virtualness  created  and  sampling  other 
communities  is  an  excellent  area  of  further  research.  Expanding  the  research  into 
other  Air  Force  communities  [such  as  maintenance  or  intelligence),  allows  for  the 
identification  of  any  cost  community  specific  differences  that  complicate  the 
implementation  of  acquisition  reform.  Ultimately,  the  goal  should  be  to  develop  a 
greater  understanding  of  the  entire  DoD.  Decision  makers  might  use  this 
information  to  structure  policies  that  overcome  the  limitations  of  virtualness  and 
effectively  improve  overall  implementation. 
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Appendix  A.  List  of  Acronyms 


ACAT 

Acquisition  Category 

AF 

Air  Force 

AFCAA 

Air  Force  Cost  Analysis  Agency 

AFMC 

Air  Force  Material  Command 

AFSPC 

Air  Force  Space  Command 

AIP 

Acquisition  Improvement  Plan 

APDP 

Acquisition  Professional  Development  Program 

CCEA 

Certified  Cost  Estimator/Analyst 

CDFM 

Certified  Defense  Financial  Management 

DoD 

Department  of  Defense 

EVM 

Earned  Value  Management 

FASA 

Federal  Acquisition  Streamlining  Act  of  1994 

FFRDC 

Federally  Funded  Research  and  Development 
Centers 

GPRA 

Government  Performance  and  Results  Act  of  1993 

LAAFB 

Los  Angeles  Air  Force  Base 

MAJCOM 

Major  Command 

MDAP 

Major  Defense  Acquisition  Program 

OSD 

Office  of  Secretary  of  Defense 

PEO 

Program  Executive  Officer 

SAR 

Selected  Acquisition  Report 

WSARA 

Weapon  Systems  Acquisition  Reform  Act  of  2009 
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Appendix  B.  Descriptive  Statistics 


Descriptive  Statistics 

N 

Minimum 

Maximum 

Mean 

Std. 

Deviation 

Skewness 

Statistic 

Error 

Kurtosis 

Statistic 

Error 

Jobsat 

102 

1.00 

5.00 

3.96 

0.82 

-0.96 

0.24 

1.03 

0.47 

LMX 

103 

1.00 

5.00 

3.87 

0.86 

-1.26 

0.24 

1.62 

0.47 

Org  Commit 

102 

1.00 

4.75 

3.29 

0.72 

-0.61 

0.24 

0.59 

0.47 

Trust 

105 

1.29 

4.71 

3.50 

0.64 

-0.76 

0.24 

1.21 

0.47 

Turnover 

Intention 

102 

1.00 

5.00 

2.11 

0.95 

0.94 

0.24 

0.33 

0.47 

Virtualness 

102 

1.06 

3.31 

2.12 

0.50 

0.50 

0.24 

-0.11 

0.47 

Reliabilities 

Trust 

(.73) 

Job  Satisfaction 

(.85) 

Leader-Member  Exchange 

(.95) 

Turnover  Intention 

(.64) 

Organizational  Commitment 

(.87) 

Virtualness 

(.79) 

Correlations 

Trust 

Jobsat 

LMX 

Turnint 

OrgCommit 

Virtualness 

Final 

Trust 

Pearson  Correlation 

1 

Sig.  (2-tailed) 

N 

105 

Jobsat 

Pearson  Correlation 

.554** 

1 

Sig.  (2-tailed) 

0 

N 

102 

102 

LMX 

Pearson  Correlation 

.777** 

.568** 

1 

Sig.  (2-tailed) 

0 

0 

N 

103 

102 

103 

Turnint 

Pearson  Correlation 

-.496** 

-.819** 

-.470** 

1 

Sig.  (2-tailed) 

0 

0 

0 

N 

102 

102 

102 

102 

OrgCommit 

Pearson  Correlation 

.556** 

.709** 

.513** 

-.760** 

1 

Sig.  (2-tailed) 

0 

0 

0 

0 

N 

102 

102 

102 

102 

102 

VirtualnessFinal 

Pearson  Correlation 

-.673** 

-.567** 

-.747** 

.443** 

-.488** 

1 

Sig.  (2-tailed) 

0 

0 

0 

0 

0 

N 

102 

102 

102 

102 

102 

102 
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Regression  Analysis  of  Hypotheses 

Variable 

Standardized 

Hypothesis 

Independent  Dependent 

3 

SE3 

Adj.  R^ 

t-value 

Sig. 

1 

Virtualness  Trust 

-0.67 

0.10 

0.45 

0.45 

-9.11 

0.00 

2 

Job  Satisfaction 

-0.57 

0.13 

0.32 

0.32 

-6.89 

0.00 

3 

Org  Commit 

-0.49 

0.13 

0.24 

0.23 

-5.59 

0.00 

4 

Turnover 

0.44 

0.17 

0.20 

0.19 

4.94 

0.00 

Intention 

Variable 

Standardized 

Hypothesis 

Independent  Dependent 

3 

SE3 

Adj.  R^ 

t-value 

Sig. 

5 

Trust  Turnover 

-0.50 

0.13 

0.25 

0.24 

-5.71 

0.00 

6 

Job  Sat  intention 

-0.82 

0.07 

0.67 

0.67 

-14.27 

0.00 

7 

Org  Commit 

-0.76 

0.09 

0.58 

0.57 

-11.70 

0.00 

Results  of  multiple  regression  analysis 

Variable 

Standardized 

Test  for 

Hypothesis 

Independent 

Dependent 

3 

SEp 

R" 

Adj. 

t-value 

Sig. 

Mediation 

8 

Trust  * 

Turnover 

-0.36 

0.17 

0.27 

0.25 

-3.10 

0.00 

2.00 

Virtualness 

Intention 

0.20 

0.22 

1.72 

0.09 

9 

Job  Sat  * 

Turnover 

-0.84 

0.08 

0.67 

0.67 

-11.96 

0.00 

3.92 

Virtualness 

Intention 

-0.03 

0.13 

-0.45 

0.65 

10 

Org  Conrimit  * 

Turnover 

-0.71 

0.10 

0.59 

0.58 

-9.62 

0.00 

3.42 

Virtualness 

Intention 

0.10 

0.14 

1.28 

0.20 
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Appendix  C.  Amos  Output  -  Structural  Equation  Model 
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Parameter  Summary  (Group  Number  1) 


Weights 

Covariances 

Variances 

Means 

Intercepts 

Total 

Fixed 

17 

0 

0 

0 

0 

17 

Labeled 

0 

0 

0 

0 

0 

0 

Unlabeled 

9 

6 

17 

0 

13 

45 

Total 

26 

6 

17 

0 

13 

62 

Result  (Default  Model) 

Minimum  was  achieved 
Chi-square  =  76.885 
Degrees  of  freedom  =  59 
Probability  level  =  .059 


Model  Fit  Summary 
CMIN 


Model 

NPAR 

CMIN 

DF 

P 

CMIN/DF 

Default  model 

45 

76.885 

59 

.059 

1.303 

Saturated  model 

104 

.000 

0 

Independence  model 

13 

633.723  91 

.000 

6.964 

Baseline  Comparisons 

Model 

NFl 

RFl 

IFl 

TLl 

CFl 

Deltal 

rhol 

Delta2 

rho2 

Default  model 

.879 

.813 

.969 

.949 

.967 

Saturated  model 

1.000 

1.000 

1.000 

Independence 

model 

.000 

.000 

.000 

.000 

.000 

Parsimony-Adjusted  Measures 


Model 

PRATIO 

PNFl 

PCFl 

Default  model 

.648 

.570 

.627 

Saturated  model 

.000 

.000 

.000 

Independence  model 

1.000 

.000 

.000 

NCP 


Model 

NCP 

L0  90 

HI  90 

Default  model 

17.885 

.000 

44.659 

Saturated  model 

.000 

.000 

.000 

Independence  model 

542.723 

466.638 

626.289 

79 


FMIN 


Model 

FMIN 

FO 

LO  90 

HI  90 

Default  model 

.725 

.169 

.000 

.421 

Saturated  model 

.000 

.000 

.000 

.000 

Independence  model 

5.979 

5.120 

4.402 

5.908 

RMSEA 


Model 

RMSEA 

LO  90 

HI  90 

PCLOSE 

Default  model 

.053 

.000 

.085 

.411 

Independence 

model 

.237 

.220 

.255 

.000 

AIC 


Model 

AIC  BCC  BIC  CAIC 

Default  model 

Saturated  model 
Independence  model 

166.885  180.580 

208.000  239.652 

659.723  663.679 

ECVI 


Model 

ECVI 

LO90 

HI  90 

MECVl 

Default  model 

1.574 

1.406 

1.827 

1.704 

Saturated  model 

1.962 

1.962 

1.962 

2.261 

Independence  model 

6.224 

5.506 

7.012 

6.261 

HOELTER 


Model 

HOELTER  HOELTER 

.05  .01 

Default  model 
Independence  model 

108  121 

20  21 
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Appendix  D.  Questionnaire 


Cost  Community  Questionnaire 

The  following  questions  pertain  to  your  current  job.  Read  each  statement  and 
answer  to  the  best  of  your  ability. 

1.  Are  you  currently  supervising  government  civilians  or  military? 

2.  Total  years  of  cost  analysis  experience  (Count  all  years  of  cost  analysis  work 
with  at  least  50%  effort  towards  cost). 

3.  Total  years  of  other  FM  experience  (Count  all  years  employed  in  FM  work  at 
least  50%  of  the  time,  including  cost  estimating  and  analysis). 

4.  Total  %  time  spent  in  acquisition  cost  estimating  during  past  six  months. 


Non-Supervisors 


Within  the  last  6  months,  what  %  Time  spent  on  following  activities: 

%Time 

a. 

%  time  spent  analyzing  Earned  Value  Management  (EVM)  data  (i.e.  CPR,  CSSR) 

using  winsight  or  other  program  in  support  of  program  management 

b. 

%  time  spent  using  Earned  Value  (EV)  data  in  support  of  creating  estimates 

c. 

%  time  spent  doing  non-cost  related  financial  management 

d. 

%  time  spent  managing  support  contractors 

e. 

%  time  consulting  with  PM  or  personnel  In  other  departments  to  discuss  and 

formulate  estimates  or  resolve  issues 

f. 

%tlme  preparing  initial  estimates 

g- 

%  time  reconciling  estimates  with  either  AFCAA  or  OSD 

h. 

%time  reviewing  estimates 

i.  %  time  completing  post  estimate  documentation 

j- 

%  time  completing  what-if  drills  for  PM  due  to  program  changes  or  budget 

k. 

%  time  spent  in  meetings 

1. 

%  time  training  (relating  to  cost  only:  included  both  time  spent  In  training  and 

time  spent  training  others) 

m. 

%  time  spent  in  non-cost  related  training  (annual  training  -  Information 

Protection,  Human  Trafficking,  etc;  do  not  include  time  spent  doing  cost  related 

n. 

%  time  Other 

Total:  Must  add  up  to  100% 

0% 

5.  You  indicated  you  spent  _%  time  completing  what-if  drills.  Please  elaborate  on 
the  reasons  for  these  drills.  Examples  of  reasons  for  what-if  drills  include 
identifying  potential  efficiencies  or  quantifying  the  impact  of  budget  changes. 
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*Trust  Items  (Schoorman  &  Ballinger,  2006) 

Read  each  statement  and,  using  the  scale  below  as  reference,  mark  the  number 
ranging  from  1  "strongly  disagree”  to  5  "strongly  agree"  which  indicates  how  you 
feel. 


1  =  Strongly  Disagree  2  =  Disagree  3  =  Neither  4  =  Agree  5  =  Strongly 
Agree 

Think  about  your  current  supervisor.  The  items  below  ask  about  your  relationship 
with,  and  thoughts  about  this  particular  individual. 

42.  My  supervisor  keeps  my  interests  in  mind  when  making  decisions. 

43. 1  would  be  willing  to  let  my  supervisor  have  complete  control  over  my  future  in 
this  organization. 

44.  If  my  supervisor  asked  why  a  problem  occurred,  1  would  speak  freely  even  if  1 
were  partly  to  blame. 

45. 1  feel  comfortable  being  creative  because  my  supervisor  understands  that 
sometimes  creative  solutions  do  not  work. 

46.  It  is  important  for  me  to  have  a  good  way  to  keep  an  eye  on  my  supervisor. 

47.  Increasing  my  vulnerability  to  criticism  by  my  supervisor  would  be  a  mistake. 

48.  If  1  had  my  way,  1  wouldn’t  let  my  supervisor  have  any  influence  over  decisions 
that  are  important  to  me. 

*Leadership  Member  Exchange  (Revised  -  Liden,  Wayne,  &  Stillwell,  1993) 

49. 1  know  where  I  stand  with  my  supervisor. 

50. 1  usually  know  how  satisfied  my  supervisor  is  with  me. 

51.  My  supervisor  understands  my  job  problems  and  needs. 

52.  My  supervisor  recognizes  my  potential. 

53.  My  supervisor  would  use  his/her  power  to  help  me  solve  work  related 
problems. 

54.  My  supervisor  would  "bail  me  out”  at  his/her  own  expense. 

55. 1  defend  and  justify  my  supervisor's  decisions  when  he/she  is  not  present  to  do 
so. 

56. 1  have  an  effective  working  relationship  with  my  supervisor. 

*Job  Satisfaction 

Read  each  statement  and,  using  the  scale  below  as  reference,  mark  the  number 
ranging  from  1  "strongly  disagree”  to  5  "strongly  agree”  which  indicates  how  you 
feel. 


1  =  Strongly  Disagree  2  =  Disagree  3  =  Neither  4  =  Agree  5  =  Strongly 
Agree 
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Think  about  your  current  employment.  The  items  below  ask  about  your  thoughts 
about  this  particular  employment. 

57.  All  in  all,  1  am  satisfied  with  my  job. 

58.  In  general,  1  don't  like  my  job. 

59.  In  general,  1  like  working  here. 

60. 1  frequently  think  of  quitting  this  job. 

*Turnover  Intentions 

61. 1  am  actively  looking  for  a  job  outside. 

62.  As  soon  as  1  can  find  a  better  job.  I'll  leave  this  organization. 

63. 1  am  seriously  thinking  about  quitting  my  job. 

64. 1  often  think  about  quitting  my  job  at  this  organization. 

65. 1  think  1  will  be  working  at  this  organization  five  years  from  now. 

Organizational  Commitment  -  affective  (Meyer  &  Allen,  1997) 

66. 1  would  be  happy  to  spend  the  rest  of  my  career  with  this  organization. 

67. 1  enjoy  discussing  my  organization  with  people  outside  it. 

68. 1  really  feel  as  if  this  organization's  problems  are  my  own. 

69. 1  think  that  1  could  easily  become  as  attached  to  another  organization  as  1  am  to 
this  one. 

70. 1  do  not  feel  like  "part  of  the  family”  at  my  organization. 

71. 1  do  not  feel  emotionally  attached  to  this  organization. 

72.  This  organization  has  a  great  deal  of  personal  meaning  for  me. 

73. 1  do  not  feel  a  strong  sense  of  belonging  to  my  organization. 

Virtualness  (Adapted  from  Carlson  &  Zmud  1999) 

Read  each  statement  and,  using  the  scale  below  as  reference,  mark  the  number 
ranging  from  1  "strongly  disagree”  to  5  "strongly  agree”  which  indicates  how  you 
feel. 


1  =  Strongly  Disagree  2  =  Disagree  3  =  Neither  4  =  Agree  5  = 

Strongly  Agree 

Think  about  your  current  supervisor.  The  items  below  ask  about  your  relationship 
with  and  thoughts  about  this  particular  individual. 

Distance 

74.  My  supervisor  thoroughly  understands  my  working  environment. 

75.  My  supervisor  works  within  close  physical  proximity  to  me. 

76.  My  team  members  understand  my  job  requirements. 

77. 1  really  understand  why  people  behave  the  way  they  do  in  my  organization. 

78. 1  often  interact  with  team  members  not  co-located  with  me. 
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79. 1  have  a  good  understanding  of  the  environment  in  which  my  organization 
operates. 

80.  Time  zones  add  difficulty  to  my  work. 

81.  My  supervisor  understands  the  daily  requirements  of  my  job. 

82.  My  supervisor  and  1  have  a  common  understanding  of  work  requirements. 

83. 1  have  a  good  sense  of  the  dynamics  within  my  organization. 

84. 1  know  what  other  people  in  my  organization  are  doing. 

Technical  Support 

85.  My  supervisor  carries  an  email-enabled  smartphone  such  as  a  Blackberry™. 

86. 1  am  unable  to  communicate  with  my  team/supervisor  without  electronics: 
telephone,  computer,  etc. 

87.  My  supervisor  is  available  by  cell  phone  throughout  the  workday. 

88. 1  primarily  communicate  with  my  team  members  through  email. 

89. 1  use  email  to  communicate  with  my  supervisor. 

90. 1  primarily  complete  my  work  through  computers. 

91. 1  contact  my  supervisor  through  telephone  conversations. 

92.  Without  computers,  1  would  be  unable  to  accomplish  my  job. 

93. 1  utilize  information  technology  in  my  daily  interactions  with  my  supervisor. 

94.  My  supervisor  carries  a  cell  phone. 

Media  Richness 

95. 1  often  speak  in  person  with  my  supervisor  to  clarify  messages  received  through 
electronic  formats. 

96.  Utilizing  email  makes  it  difficult  to  understand  the  tone  of  messages  from  my 
supervisor. 

97. 1  am  easily  able  to  understand  a  variety  of  different  cues  (e.g.,  emotional  tone, 
feelings]  from  my  supervisor. 

98. 1  often  seek  instructions  sent  via  email  from  my  supervisor  regarding  work 
requirements. 

99. 1  am  easily  able  to  understand  the  message  from  my  supervisor. 

100.  1  am  easily  able  to  tailor  my  messages  to  my  supervisor. 

101.  1  often  speak  in  person  with  my  coworkers  to  clarify  messages  received 
through  electronic  formats. 

102.  1  am  able  to  use  rich  and  varied  language  when  communicating  with  my 
supervisor. 

103.  It  is  easy  to  exchange  timely  feedback  with  my  supervisor. 

104.  1  am  easily  able  to  maintain  multiple  conversations  with  co¬ 
workers/supervisors. 

Synchronicity 

105.  It  negatively  affects  my  work  when  my  supervisor  is  absent. 

106.  My  supervisor  is  available  whenever  1  need  him/her. 

107.  My  supervisor’s  work  schedule  is  in-sync  with  my  own  work  schedule. 
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108.  My  supervisor  answers  my  questions  on  the  same  day  I  send  the  email. 

109.  It  is  often  difficult  to  get  in  touch  with  my  supervisor. 

110.  My  supervisor  and  I  often  have  misunderstandings  driven  by  the  differences 
in  our  schedules. 

111.  My  supervisor  and  I  have  difficulties  aligning  our  schedules. 

112.  My  schedule  changes  based  on  my  supervisor's  schedule. 

113.  My  supervisor  and  I  always  seem  to  be  in  tune  as  to  what  we  are  doing. 

114.  My  supervisor  responds  to  my  messages  [e.g.,  phone,  email]  in  a  timely 
manner. 

115.  Demographics 

116.  AF  Organization/  Office  symbol: 

117.  ACAT  Program[s]  worked  on  during  past  6  months: 

118.  Status:  Full-time,  Part-time 

119.  Current  Mil  Grade  or  Civilian  Pay  Plan-Series-Grade: 

120.  How  far  away  in  miles  are  you  located  from  your  supervisor? 

121.  Name  of  supervisor: 

122.  DutyAFSC: 

123.  ADPD  Certification  type  and  level: 

124.  Financial  certificates:  CDFM,  CDFM-A,  CPA,  CCEA,  other 

125.  Duty  title: 

126.  Full  name: 
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Supervisors 

1.  How  many  acquisition  programs  and  what  category  do  you  currently  manage? 

2.  Explain  whether  you  have  enough  cost  analysts  to  effectively  support  your 
team's  portfolio  of  ACAT  programs.  Include  the  number  of  extra  or  needed 
analysts  and  what  programs  they  will  support. 

a.  Explain  any  tasks  that  are  being  sacrificed  in  order  to  accomplish  the 
mission. 

b.  Explain  any  tasks  that  are  being  completed  that  do  not  contribute  to  the 
mission. 

3.  How  many  cost  estimators,  over  how  many  months,  does  it  take  to  perform  a 
thorough  estimate  of  the  cost  of  a  $500-million  new  program? 

4.  Is  there  a  disparity  between  what  your  team  is  presently  working  on  and  what 
you  prefer  your  team  to  be  accomplishing?  Explain. 

5.  Do  you  have  sufficient  access  to  resources  such  as  data? 

6.  Do  you  have  a  sufficient  support  network  of  subject  matter  experts? 

7.  Do  you  receive  adequate  support  from  AFCAA? 

8.  Are  you  satisfied  with  the  technical  competencies  of  your  cost  analysis  staff? 

9.  Currently,  what  kind  of  technical  skill,  experience,  knowledge  or  other 
competencies  would  you  say  are  generally  lacking  or  not  available  among  the 
cost  analysts  in  your  group? 

10.  What  would  get  done  that  is  not  now  done  if  your  cost  analysts  had  these 
competencies? 

11.  What  would  get  done  better  if  your  cost  analysts  had  these  competencies? 

12.  Looking  five  years  ahead,  what  kind  of  different  technical  skills,  experience, 
knowledge,  or  competencies  may  be  needed? 

13.  How  often  does  your  team  change,  either  in  actual  personnel  or  activities? 

14.  How  many  cost  analysts  were  hired  specifically  to  work  on  your  group’s 
programs  over  the  past  24  months? 

15.  How  difficult  is  it  to  recruit  cost  analysts  in  your  area? 

16.  Is  there  a  difference  in  recruiting  government  civilians  versus  contractors? 

17.  How  many  cost  analysts  left  your  group  in  the  past  24  months? 

18.  What  is  your  understanding  of  why  they  left  and  where  they  went? 

19.  What  measures  would  you  suggest  taking  to  improve  the  training  and 
performance  of  cost  analysts? 

20.  How  do  you  think  the  newly  developed  APDP  cost  certification  (BUS-CE] 
program  is  going  to  address  some  of  the  training  deficiencies? 

21.  What  steps  have  you  taken  to  support  recently  implemented  acquisition  policies, 
specifically  Weapon  Systems  Acquisition  Reform  Act  (WSARA)? 

22.  Do  you  feel  WSARA  addresses  the  deficiencies  of  the  acquisition  community? 
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a.  If  not,  what  deficiencies  still  need  addressing? 

23.  Have  any  of  the  changes  implemented  to  support  WSARA  been  detrimental  to 
your  organization? 

24.  Have  any  of  the  changes  implemented  to  support  WSARA  been  beneficial  to  your 
organization? 


Number  employed  of  each 

Civilian 

Military 

Contractor 

FFRDC 

Total 

Within  the  last  6  months,  what  %  Time  spent  on  following  activities: 

%  Time 

%  Time 

%  Time 

%  Time 

%  Time 

a.  %  time  spent  analyzing  Earned  Value  Management  (EVM)  data  (i.e.  CPR,  CSSR) 
using  winsight  or  other  program  in  support  of  program  management 

b.  %  time  spent  using  Earned  Value  (EV)  data  in  support  of  creating  estimates 

c.  %  time  spent  doing  non-cost  related  financial  management 

d.  %  time  spent  managing  support  contractors 

e.  %  time  consulting  with  PM  or  personnel  in  other  departments  to  discuss  and 
formulate  estimates  or  resolve  issues 

f.  %time  preparing  initial  estimates 

g.  %  time  reconciling  estimates  with  either  AFCAA  or  OSD 

h.  %  time  reviewing  estimates 

i.  %  time  completing  post  estimate  documentation 

j.  %  time  completing  what-if  drills  for  PM  due  to  program  changes  or  budget 

k.  %  time  spent  in  meetings 

1.  %  time  training  (relating  to  cost  only:  included  both  time  spent  in  training  and 

time  spent  training  others) 

m.  %  time  spent  in  non-cost  related  training  (annual  training  -  Information 
Protection,  Human  Trafficking,  etc;  do  not  include  time  spent  doing  cost  related 

n.  %  time  Other 

Total:  Must  add  up  to  100% 

0% 

0% 

0% 

0% 

25.  You  indicated  you  spent  _%  time  completing  what-if  drills.  Please  elaborate  on 
the  reasons  for  these  drills.  Examples  of  reasons  for  what-if  drills  include 
identifying  potential  efficiencies  or  quantifying  the  impact  of  budget  changes. 


Read  each  statement  and,  using  the  scale  below  as  reference,  mark  the  number 
ranging  from  1  "strongly  disagree”  to  5  "strongly  agree"  which  indicates  how  you 
feel. 


1  =  Strongly  Disagree  2  =  Disagree  3  =  Neither  4  =  Agree  5  = 

Strongly  Agree 

Think  about  your  current  subordinates.  The  items  below  ask  about  your 
relationship  with  and  thoughts  about  those  particular  individuals. 

26. 1  am  easily  able  to  understand  the  message  from  my  subordinates. 

27. 1  am  able  to  use  rich  and  varied  language  when  communicating  with  my 
subordinates. 

28.  My  subordinates  and  I  often  have  misunderstandings  driven  by  the  differences 
in  our  schedule. 

29.  My  subordinates  thoroughly  understand  my  working  environment. 

30.  My  subordinates  understand  the  daily  requirements  of  my  job. 
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31.  My  subordinates  answer  my  questions  on  the  same  day  I  send  the  email. 

32.  My  schedule  changes  based  on  my  subordinates’  schedule. 

33.  My  subordinates  work  within  close  physical  proximity  to  me. 

34. 1  use  email  to  communicate  with  my  subordinates. 

35.  Without  computers,  1  would  be  unable  to  accomplish  my  job. 

36. 1  primarily  complete  my  work  through  computers. 

37. 1  have  a  good  sense  of  the  dynamics  within  my  organization. 

38.  It  is  often  difficult  to  get  in  touch  with  my  subordinates. 

39.  Utilizing  email  makes  it  difficult  to  understand  the  tone  of  messages  from  my 
subordinates. 

40.  My  subordinates’  work  schedules  are  in-sync  with  my  own  work  schedule. 

41.  My  subordinates  and  1  have  difficulties  aligning  our  schedules. 

42. 1  know  what  other  people  in  my  organization  are  doing. 

43.  My  team  members  understand  my  job  requirements. 

44. 1  often  speak  in  person  with  my  coworkers  to  clarify  messages  received  through 
electronic  formats. 

45.  My  subordinates  respond  to  my  messages  (e.g. ,  phone,  email)  in  a  timely 
manner. 

46.  Time  zones  add  difficulty  to  my  work. 

47. 1  primarily  communicate  with  my  team  members  through  email. 

48. 1  often  speak  in  person  with  my  subordinates  to  clarify  messages  received 
through  electronic  formats. 

49.  My  subordinates  and  1  always  seem  to  be  in  tune  as  to  what  we  are  doing. 

50.  It  is  easy  to  exchange  timely  feedback  with  my  subordinates. 

51.  My  subordinates  are  available  by  cell  phone  throughout  the  workday. 

52. 1  contact  my  subordinates  through  telephone  conversations. 

53. 1  am  easily  able  to  understand  a  variety  of  different  cues  [e.g..  emotional  tone, 
feelings]  from  my  subordinates. 

54. 1  have  a  good  understanding  of  the  environment  in  which  my  organization 
operates. 

55. 1  really  understand  why  people  behave  the  way  they  do  in  my  organization. 

56. 1  am  easily  able  to  maintain  multiple  conversations  with  subordinates. 

57.  My  subordinates  carry  a  cell  phone. 

58.  My  subordinates  carry  an  email-enabled  smartphone  such  as  a  Blackberry™. 

59. 1  am  easily  able  to  tailor  my  messages  to  my  subordinates. 

60.  My  subordinates  are  available  whenever  1  need  them. 

61.  My  subordinates  and  1  have  a  common  understanding  of  work  requirements. 

62. 1  utilize  information  technology  in  my  daily  interactions  with  my  subordinates. 
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63. 1  am  unable  to  communicate  with  my  team/  subordinates  without  electronics: 
telephone,  computer,  etc. 

64. 1  often  interact  with  team  members  not  co-located  with  me. 

65. 1  often  seek  email  clarifications  from  my  subordinates  regarding  work  requests. 

66.  It  negatively  affects  my  work  when  my  subordinates  are  absent. 

To  be  completed  Per  Individual  Supervised 

67.  Name  of  subordinate: 

Leader  Appraisal  of  the  Member's  Performance  (Liden,  Wayne  &  Stillwell, 
1997) 

Read  each  statement  and,  using  the  scale  below  as  reference,  mark  the  number 
ranging  from  1  "very  ineffective"  to  5  "very  effective"  which  indicates  how  you  feel. 

1  =  Very  Ineffective  2  =  Ineffective  3  =  Neither  4  =  Effective  5  =  Very 
Effective 


Think  about  your  current  subordinate.  The  items  below  ask  about  your  relationship 
with,  and  thoughts  about,  this  particular  individual. 

68.  Rate  the  overall  level  of  performance  you  observe  for  this  subordinate. 

69.  What  is  your  personal  view  of  your  subordinate  in  terms  of  overall 
effectiveness? 

70.  Overall,  to  what  extent  do  you  feel  your  subordinate  effectively  fulfills  his  or  her 
role  and  responsibilities? 

Read  each  statement  and,  using  the  scale  below  as  reference,  mark  the  number 
ranging  from  1  "strongly  disagree"  to  5  "strongly  agree"  which  indicates  how  you 
feel. 


1  =  Strongly  Disagree  2  =  Disagree  3  =  Neither  4  =  Agree  5  = 

Strongly  Agree 

Think  about  your  current  subordinate.  The  items  below  ask  about  your  relationship 
with  and  thoughts  about  this  particular  individual. 

71.  This  subordinate  is  superior  to  other  subordinates  1  have  supervised  before. 

Trust  (Mayer  &  Davis,  1999) 

72.  My  subordinate  keeps  my  interests  in  mind  when  making  decisions. 

73.  If  my  subordinate  asked  why  a  problem  occurred,  1  would  speak  freely  even  if  1 
were  partly  to  blame. 
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74.  It  is  important  for  me  to  have  a  good  way  to  keep  an  eye  on  my  subordinate. 

75.  Increasing  my  vulnerability  to  criticism  by  my  subordinate  would  be  a  mistake. 

76.  If  I  had  my  way,  I  wouldn’t  let  my  subordinate  have  any  influence  over  decisions 
that  are  important  to  me. 

Demographics 

77.  AF  Organization/  Office  symbol: 

78.  ACAT  Programfs]  worked  on  during  past  6  months: 

79.  Status:  Full-time,  Part-time 

80.  Current  Mil  Grade  or  Civilian  Pay  Plan-Series-Grade: 

81.  How  far  away  in  miles  are  you  located  from  your  supervisor? 

82.  Duty  AFSC: 

83.  ADPD  Certification  type  and  level: 

84.  Financial  certificates:  CDFM,  CDFM-A,  CCEA 

85.  Duty  title: 

86.  Full  name: 
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Appendix  E.  Questionnaire  Pivot  Tables 
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Variation  between  Supervisors  and  Organic  Employees 
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